2024-09-22T03:24:40.009 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T03:24:40.027 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:24:40.113 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914779 branch: squid description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_omap_write} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914779' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid env: CRIMSON_COMPAT: '1' sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1158 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGJYukSc47m/qSHkVP+WSf278IHS1zf2QuK7xT9+NhTNU5d1nIt/KgAYwNztTICj6V80R1CcJlJ/Th2u8RrF+U= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - radosbench: clients: - client.0 objectsize: 4096 size: 4096 time: 300 write-omap: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T03:24:40.114 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-22T03:24:40.114 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-22T03:24:40.114 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T03:24:40.116 INFO:teuthology.task.internal:Checking packages... 2024-09-22T03:24:40.134 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-22T03:24:40.135 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T03:24:40.135 INFO:teuthology.packaging:ref: None 2024-09-22T03:24:40.135 INFO:teuthology.packaging:tag: None 2024-09-22T03:24:40.135 INFO:teuthology.packaging:branch: squid 2024-09-22T03:24:40.135 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:24:40.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-22T03:24:40.334 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-22T03:24:40.336 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T03:24:40.342 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T03:24:40.342 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T03:24:40.619 INFO:teuthology.task.internal:Saving configuration 2024-09-22T03:24:40.634 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T03:24:40.640 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T03:24:40.665 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914779', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 03:19:30.775168', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGJYukSc47m/qSHkVP+WSf278IHS1zf2QuK7xT9+NhTNU5d1nIt/KgAYwNztTICj6V80R1CcJlJ/Th2u8RrF+U='} 2024-09-22T03:24:40.665 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T03:24:40.671 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-22T03:24:40.671 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T03:24:40.705 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f816d1f0160>, signals=[15]) 2024-09-22T03:24:40.705 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T03:24:40.711 INFO:teuthology.task.internal:Opening connections... 2024-09-22T03:24:40.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:24:40.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:24:40.785 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T03:24:40.791 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-09-22T03:24:40.864 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-09-22T03:24:40.864 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-09-22T03:24:40.921 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-09-22T03:24:40.921 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-09-22T03:24:40.921 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-09-22T03:24:40.921 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-09-22T03:24:40.921 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-09-22T03:24:40.921 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-09-22T03:24:40.921 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T03:24:40.921 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-09-22T03:24:40.922 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-09-22T03:24:40.922 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T03:24:40.922 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-09-22T03:24:40.922 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T03:24:40.922 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T03:24:40.922 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T03:24:40.923 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-09-22T03:24:40.964 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T03:24:40.973 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T03:24:40.978 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T03:24:40.978 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-09-22T03:24:40.994 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T03:24:41.001 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T03:24:41.001 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-09-22T03:24:41.125 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T03:24:41.158 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T03:24:41.172 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T03:24:41.172 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T03:24:41.173 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T03:24:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:24:41.190 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-22T03:24:41.248 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-503.el9.x86_64 2024-09-22T03:24:41.248 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-503.el9.x86_64 2024-09-22T03:24:41.248 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-09-22T03:24:44.941 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-22T03:24:46.742 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-22T03:24:50.935 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-22T03:24:52.578 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-22T03:24:53.999 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-22T03:24:59.355 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-09-22T03:25:01.261 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T03:25:01.261 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T03:25:01.351 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T03:25:01.363 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T03:25:01.364 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:25:01.364 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:25:01.364 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:25:01.364 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-09-22T03:25:01.364 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:25:01.364 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 79 M 2024-09-22T03:25:01.364 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 126 M 2024-09-22T03:25:01.364 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:25:01.515 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 276 kB/s | 36 kB 00:00 2024-09-22T03:25:04.690 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 10 MB/s | 35 MB 00:03 2024-09-22T03:25:08.156 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-09-22T03:25:08.548 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 4.1 MB/s | 29 MB 00:07 2024-09-22T03:25:08.548 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:25:08.548 INFO:teuthology.orchestra.run.smithi122.stdout:Total 11 MB/s | 79 MB 00:07 2024-09-22T03:25:09.230 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:25:09.570 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:25:09.571 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:25:10.541 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:25:10.541 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:25:12.979 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:25:13.686 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:25:13.769 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:25:16.536 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:25:16.670 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:25:21.471 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:25:21.570 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:25:26.432 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:26:04.646 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:26:04.685 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:26:05.015 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:26:05.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:26:05.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:26:05.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:26:05.884 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:26:06.240 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:26:06.325 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:26:06.326 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:26:06.326 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T03:26:06.326 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T03:26:06.327 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T03:26:06.327 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T03:26:06.327 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T03:26:06.327 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:26:06.385 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:26:06.385 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:26:06.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-09-22T03:26:06.417 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 82.el9 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8274670 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T03:26:06.418 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:26:06.419 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T03:26:06.420 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T03:26:06.420 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-508.el9.x86_64 2024-09-22T03:26:06.420 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T03:26:07.096 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-09-22T03:26:08.418 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T03:26:08.449 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-09-22T03:26:08.456 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-22T03:26:08.471 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.yuABPXWjOA 2024-09-22T03:26:08.472 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yuABPXWjOA 2024-09-22T03:26:08.539 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.yuABPXWjOA 2024-09-22T03:26:08.678 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.yuABPXWjOA is 6KB 2024-09-22T03:26:08.726 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.yuABPXWjOA 2024-09-22T03:26:08.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T03:26:08.805 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T03:26:08.805 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T03:26:08.805 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T03:26:08.805 INFO:teuthology.orchestra.run.smithi122.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T03:26:08.806 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T03:26:08.955 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-09-22T03:26:38.964 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T03:26:38.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-22T03:26:38.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:27:39.023 DEBUG:teuthology.orchestra.remote:timed out 2024-09-22T03:27:48.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-22T03:27:48.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:27:51.535 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-22T03:27:52.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-22T03:27:52.003 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:27:52.003 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-09-22T03:27:52.017 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:27:52.017 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:27:52.017 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:27:52.017 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:27:53.019 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T03:27:53.019 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:27:53.116 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-09-22T03:27:53.143 DEBUG:teuthology.parallel:result is None 2024-09-22T03:27:53.143 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T03:27:53.150 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T03:27:53.150 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:27:53.164 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T03:27:53.170 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T03:27:53.176 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T03:27:53.176 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:27:53.231 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T03:27:53.237 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T03:27:53.238 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:27:53.299 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:27:53.309 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:27:53.311 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T03:27:53.318 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T03:27:53.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:27:53.377 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T03:27:53.385 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T03:27:53.385 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:27:53.433 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:27:53.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:27:53.559 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:27:53.641 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:27:53.681 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:27:53.681 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:27:53.746 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-09-22T03:27:53.841 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:27:54.106 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T03:27:54.116 INFO:teuthology.task.internal:Starting timer... 2024-09-22T03:27:54.117 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T03:27:54.151 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T03:27:54.183 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-22T03:27:54.183 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-09-22T03:27:54.224 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-09-22T03:27:54.226 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T03:27:54.226 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-09-22T03:27:54.287 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-09-22T03:27:54.287 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-09-22T03:27:54.287 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:27:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:27:54.352 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T03:27:54.352 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:27:54.437 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T03:27:54.446 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T03:27:54.456 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T03:27:54.457 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T03:31:35.459 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-09-22T03:31:35.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-22T03:31:35.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:31:35.534 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-09-22T03:31:35.617 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-09-22T03:31:35.617 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T03:31:35.627 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T03:31:35.627 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T03:31:35.628 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:31:35.686 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T03:31:35.700 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T03:31:35.724 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-09-22T03:31:35.735 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-09-22T03:31:35.749 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T03:31:35.763 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T03:31:35.817 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:31:35.821 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:31:35.821 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-09-22T03:31:35.821 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:31:35.821 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:31:35.821 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:31:35.821 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:31:35.822 INFO:teuthology.run_tasks:Running task install... 2024-09-22T03:31:35.830 DEBUG:teuthology.task.install:project ceph 2024-09-22T03:31:35.831 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-22T03:31:35.831 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'} 2024-09-22T03:31:35.831 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T03:31:35.836 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T03:31:35.836 INFO:teuthology.task.install:extra packages: [] 2024-09-22T03:31:35.836 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-22T03:31:35.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:31:35.979 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/ 2024-09-22T03:31:35.980 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-22T03:31:36.089 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T03:31:36.089 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:31:36.089 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T03:31:36.115 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T03:31:36.115 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T03:31:36.182 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T03:31:36.257 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T03:31:36.321 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-09-22T03:31:36.323 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-09-22T03:31:36.558 INFO:teuthology.orchestra.run.smithi122.stdout:45 files removed 2024-09-22T03:31:36.584 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-09-22T03:31:37.147 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 313 kB/s | 87 kB 00:00 2024-09-22T03:31:37.415 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 52 kB/s | 12 kB 00:00 2024-09-22T03:31:37.674 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 8.9 kB/s | 2.0 kB 00:00 2024-09-22T03:31:38.265 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.3 MB 00:00 2024-09-22T03:31:40.025 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-22T03:31:45.592 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-22T03:31:47.035 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-22T03:31:48.639 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 15 MB/s | 23 MB 00:01 2024-09-22T03:31:54.079 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-22T03:31:56.296 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:31:56.319 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 9.9 M 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 19 M 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T03:31:56.320 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 760 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 501 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.1 M 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-22T03:31:56.321 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 161 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 301 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:31:56.322 INFO:teuthology.orchestra.run.smithi122.stdout:Install 30 Packages 2024-09-22T03:31:56.323 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-09-22T03:31:56.323 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:31:56.323 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 58 M 2024-09-22T03:31:56.323 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:31:57.263 INFO:teuthology.orchestra.run.smithi122.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 8.9 MB/s | 5.4 MB 00:00 2024-09-22T03:31:57.313 INFO:teuthology.orchestra.run.smithi122.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 504 kB/s | 25 kB 00:00 2024-09-22T03:31:57.422 INFO:teuthology.orchestra.run.smithi122.stdout:(3/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 13 MB/s | 9.9 MB 00:00 2024-09-22T03:31:57.456 INFO:teuthology.orchestra.run.smithi122.stdout:(4/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 5.2 MB/s | 760 kB 00:00 2024-09-22T03:31:57.481 INFO:teuthology.orchestra.run.smithi122.stdout:(5/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 8.4 MB/s | 501 kB 00:00 2024-09-22T03:31:57.539 INFO:teuthology.orchestra.run.smithi122.stdout:(6/32): python3-ceph-argparse-19.1.1-373.g4e45e 774 kB/s | 45 kB 00:00 2024-09-22T03:31:57.615 INFO:teuthology.orchestra.run.smithi122.stdout:(7/32): python3-ceph-common-19.1.1-373.g4e45e6b 1.8 MB/s | 136 kB 00:00 2024-09-22T03:31:57.715 INFO:teuthology.orchestra.run.smithi122.stdout:(8/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_64 20 MB/s | 5.1 MB 00:00 2024-09-22T03:31:57.749 INFO:teuthology.orchestra.run.smithi122.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 1.2 MB/s | 161 kB 00:00 2024-09-22T03:31:57.832 INFO:teuthology.orchestra.run.smithi122.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 2.7 MB/s | 321 kB 00:00 2024-09-22T03:31:57.866 INFO:teuthology.orchestra.run.smithi122.stdout:(11/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 2.5 MB/s | 301 kB 00:00 2024-09-22T03:31:57.891 INFO:teuthology.orchestra.run.smithi122.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-22T03:31:58.100 INFO:teuthology.orchestra.run.smithi122.stdout:(13/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x 14 MB/s | 19 MB 00:01 2024-09-22T03:31:58.717 INFO:teuthology.orchestra.run.smithi122.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 73 kB/s | 45 kB 00:00 2024-09-22T03:31:58.742 INFO:teuthology.orchestra.run.smithi122.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 122 kB/s | 107 kB 00:00 2024-09-22T03:31:58.834 INFO:teuthology.orchestra.run.smithi122.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 170 kB/s | 160 kB 00:00 2024-09-22T03:31:59.451 INFO:teuthology.orchestra.run.smithi122.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 68 kB/s | 42 kB 00:00 2024-09-22T03:31:59.651 INFO:teuthology.orchestra.run.smithi122.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 322 kB/s | 292 kB 00:00 2024-09-22T03:31:59.702 INFO:teuthology.orchestra.run.smithi122.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 674 kB/s | 662 kB 00:00 2024-09-22T03:32:00.235 INFO:teuthology.orchestra.run.smithi122.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 61 kB/s | 36 kB 00:00 2024-09-22T03:32:00.260 INFO:teuthology.orchestra.run.smithi122.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 273 kB/s | 221 kB 00:00 2024-09-22T03:32:00.285 INFO:teuthology.orchestra.run.smithi122.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 82 kB/s | 47 kB 00:00 2024-09-22T03:32:00.353 INFO:teuthology.orchestra.run.smithi122.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 506 kB/s | 26 kB 00:00 2024-09-22T03:32:00.386 INFO:teuthology.orchestra.run.smithi122.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-22T03:32:00.412 INFO:teuthology.orchestra.run.smithi122.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 829 kB/s | 49 kB 00:00 2024-09-22T03:32:00.437 INFO:teuthology.orchestra.run.smithi122.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-22T03:32:00.512 INFO:teuthology.orchestra.run.smithi122.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-22T03:32:00.696 INFO:teuthology.orchestra.run.smithi122.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.9 MB/s | 839 kB 00:00 2024-09-22T03:32:00.896 INFO:teuthology.orchestra.run.smithi122.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 4.1 MB/s | 1.6 MB 00:00 2024-09-22T03:32:01.321 INFO:teuthology.orchestra.run.smithi122.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-09-22T03:32:01.380 INFO:teuthology.orchestra.run.smithi122.stdout:(31/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 4.7 MB/s | 3.2 MB 00:00 2024-09-22T03:32:02.163 INFO:teuthology.orchestra.run.smithi122.stdout:(32/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 2.5 MB/s | 3.1 MB 00:01 2024-09-22T03:32:02.168 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:32:02.168 INFO:teuthology.orchestra.run.smithi122.stdout:Total 9.8 MB/s | 58 MB 00:05 2024-09-22T03:32:02.276 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:32:02.311 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:32:02.312 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:32:02.979 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:32:02.979 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:32:04.441 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:32:04.853 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-22T03:32:04.915 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:32:05.008 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:32:05.106 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:32:05.134 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:32:05.217 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:32:05.246 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-09-22T03:32:05.271 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-09-22T03:32:05.294 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:32:05.331 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:32:05.531 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:32:05.567 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:32:05.596 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-09-22T03:32:05.640 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 10/34 2024-09-22T03:32:05.669 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:32:05.746 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-09-22T03:32:05.782 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:32:05.886 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:32:05.931 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-09-22T03:32:06.043 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-22T03:32:06.074 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-22T03:32:06.115 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-22T03:32:06.156 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-22T03:32:06.185 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-22T03:32:06.221 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-22T03:32:06.568 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-22T03:32:06.613 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:32:06.688 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:32:06.719 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-22T03:32:06.762 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-22T03:32:07.253 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-22T03:32:07.283 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:32:07.371 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:32:07.448 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-22T03:32:07.477 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-22T03:32:09.758 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:32:09.798 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:32:10.371 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:32:10.426 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:32:10.520 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:32:10.520 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T03:32:10.520 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T03:32:10.520 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:10.553 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:32:20.692 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:32:20.692 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-09-22T03:32:20.692 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-09-22T03:32:20.692 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-09-22T03:32:20.693 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-09-22T03:32:20.693 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-09-22T03:32:20.693 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-09-22T03:32:20.693 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-09-22T03:32:20.693 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:21.688 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:32:21.713 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:32:21.713 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:32:21.713 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T03:32:21.714 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:32:21.714 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:32:21.714 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:21.715 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:32:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:32:21.799 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:32:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:32:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-22T03:32:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:32:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:32:22.999 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:32:23.000 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-22T03:32:23.001 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-22T03:32:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-22T03:32:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-22T03:32:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-22T03:32:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-22T03:32:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-22T03:32:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:32:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-22T03:32:23.002 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-22T03:32:23.998 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:32:23.998 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:23.998 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-09-22T03:32:23.998 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T03:32:23.999 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:24.000 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T03:32:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T03:32:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T03:32:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T03:32:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T03:32:24.001 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:24.001 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:32:24.316 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-09-22T03:32:24.902 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:30 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:32:25.034 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:32:25.034 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:32:25.034 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 124 M 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout: libasan x86_64 11.5.0-2.el9 appstream 414 k 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 158 k 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:32:25.035 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:32:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:Install 7 Packages 2024-09-22T03:32:25.036 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 125 M 2024-09-22T03:32:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.1 G 2024-09-22T03:32:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:32:25.563 INFO:teuthology.orchestra.run.smithi122.stdout:(1/7): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 759 kB/s | 158 kB 00:00 2024-09-22T03:32:25.630 INFO:teuthology.orchestra.run.smithi122.stdout:(2/7): libasan-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 414 kB 00:00 2024-09-22T03:32:25.689 INFO:teuthology.orchestra.run.smithi122.stdout:(3/7): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-09-22T03:32:25.739 INFO:teuthology.orchestra.run.smithi122.stdout:(4/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-09-22T03:32:25.798 INFO:teuthology.orchestra.run.smithi122.stdout:(5/7): libubsan-11.5.0-2.el9.x86_64.rpm 781 kB/s | 183 kB 00:00 2024-09-22T03:32:25.831 INFO:teuthology.orchestra.run.smithi122.stdout:(6/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.3 MB/s | 122 kB 00:00 2024-09-22T03:32:30.923 INFO:teuthology.orchestra.run.smithi122.stdout:(7/7): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 22 MB/s | 124 MB 00:05 2024-09-22T03:32:30.926 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:32:30.926 INFO:teuthology.orchestra.run.smithi122.stdout:Total 21 MB/s | 125 MB 00:05 2024-09-22T03:32:30.980 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:32:31.000 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:32:31.000 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:32:31.825 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:32:31.825 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:32:32.912 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:32:32.971 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-09-22T03:32:33.024 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-09-22T03:32:33.062 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-09-22T03:32:33.099 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:32:33.140 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libasan-11.5.0-2.el9.x86_64 5/7 2024-09-22T03:32:33.168 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:32:50.348 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:32:50.431 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:32:59.057 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:32:59.057 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/7 2024-09-22T03:32:59.057 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/7 2024-09-22T03:32:59.057 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libasan-11.5.0-2.el9.x86_64 3/7 2024-09-22T03:32:59.057 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:32:59.057 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-09-22T03:32:59.057 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-09-22T03:32:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-09-22T03:32:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:59.410 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:32:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:59.410 INFO:teuthology.orchestra.run.smithi122.stdout: libasan-11.5.0-2.el9.x86_64 2024-09-22T03:32:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:32:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T03:32:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T03:32:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T03:32:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T03:32:59.411 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:32:59.411 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:32:59.718 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-09-22T03:33:00.295 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:06 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:33:00.443 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.3 M 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-22T03:33:00.446 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.9 M 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 120 M 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T03:33:00.447 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T03:33:00.448 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout:Install 38 Packages 2024-09-22T03:33:00.449 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:00.450 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 134 M 2024-09-22T03:33:00.450 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.4 G 2024-09-22T03:33:00.450 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:33:01.017 INFO:teuthology.orchestra.run.smithi122.stdout:(1/38): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 25 kB/s | 6.6 kB 00:00 2024-09-22T03:33:01.326 INFO:teuthology.orchestra.run.smithi122.stdout:(2/38): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 2.6 MB/s | 1.5 MB 00:00 2024-09-22T03:33:01.451 INFO:teuthology.orchestra.run.smithi122.stdout:(3/38): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 3.3 MB/s | 2.3 MB 00:00 2024-09-22T03:33:01.552 INFO:teuthology.orchestra.run.smithi122.stdout:(4/38): ceph-mgr-modules-core-19.1.1-373.g4e45e 2.4 MB/s | 249 kB 00:00 2024-09-22T03:33:01.752 INFO:teuthology.orchestra.run.smithi122.stdout:(5/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-22T03:33:01.869 INFO:teuthology.orchestra.run.smithi122.stdout:(6/38): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-09-22T03:33:01.931 INFO:teuthology.orchestra.run.smithi122.stdout:(7/38): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 106 kB 00:00 2024-09-22T03:33:01.981 INFO:teuthology.orchestra.run.smithi122.stdout:(8/38): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-22T03:33:02.031 INFO:teuthology.orchestra.run.smithi122.stdout:(9/38): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-22T03:33:02.090 INFO:teuthology.orchestra.run.smithi122.stdout:(10/38): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-22T03:33:02.257 INFO:teuthology.orchestra.run.smithi122.stdout:(11/38): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-22T03:33:02.382 INFO:teuthology.orchestra.run.smithi122.stdout:(12/38): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-09-22T03:33:02.433 INFO:teuthology.orchestra.run.smithi122.stdout:(13/38): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-22T03:33:02.475 INFO:teuthology.orchestra.run.smithi122.stdout:(14/38): python3-markupsafe-1.1.1-12.el9.x86_64 830 kB/s | 35 kB 00:00 2024-09-22T03:33:02.517 INFO:teuthology.orchestra.run.smithi122.stdout:(15/38): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-22T03:33:02.559 INFO:teuthology.orchestra.run.smithi122.stdout:(16/38): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-22T03:33:02.600 INFO:teuthology.orchestra.run.smithi122.stdout:(17/38): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-22T03:33:02.884 INFO:teuthology.orchestra.run.smithi122.stdout:(18/38): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-09-22T03:33:02.935 INFO:teuthology.orchestra.run.smithi122.stdout:(19/38): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-22T03:33:03.001 INFO:teuthology.orchestra.run.smithi122.stdout:(20/38): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_ 2.5 MB/s | 4.9 MB 00:01 2024-09-22T03:33:03.018 INFO:teuthology.orchestra.run.smithi122.stdout:(21/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-09-22T03:33:03.102 INFO:teuthology.orchestra.run.smithi122.stdout:(22/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-22T03:33:03.119 INFO:teuthology.orchestra.run.smithi122.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 643 kB/s | 11 kB 00:00 2024-09-22T03:33:03.144 INFO:teuthology.orchestra.run.smithi122.stdout:(24/38): python3-cherrypy-18.6.1-2.el9.noarch.r 2.9 MB/s | 358 kB 00:00 2024-09-22T03:33:03.169 INFO:teuthology.orchestra.run.smithi122.stdout:(25/38): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-22T03:33:03.195 INFO:teuthology.orchestra.run.smithi122.stdout:(26/38): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-22T03:33:03.220 INFO:teuthology.orchestra.run.smithi122.stdout:(27/38): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-22T03:33:03.245 INFO:teuthology.orchestra.run.smithi122.stdout:(28/38): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-22T03:33:03.271 INFO:teuthology.orchestra.run.smithi122.stdout:(29/38): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-22T03:33:03.304 INFO:teuthology.orchestra.run.smithi122.stdout:(30/38): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-22T03:33:03.338 INFO:teuthology.orchestra.run.smithi122.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 491 kB/s | 16 kB 00:00 2024-09-22T03:33:03.388 INFO:teuthology.orchestra.run.smithi122.stdout:(32/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-22T03:33:03.413 INFO:teuthology.orchestra.run.smithi122.stdout:(33/38): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-22T03:33:03.439 INFO:teuthology.orchestra.run.smithi122.stdout:(34/38): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-22T03:33:03.514 INFO:teuthology.orchestra.run.smithi122.stdout:(35/38): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-22T03:33:03.548 INFO:teuthology.orchestra.run.smithi122.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 601 kB/s | 20 kB 00:00 2024-09-22T03:33:03.590 INFO:teuthology.orchestra.run.smithi122.stdout:(37/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-09-22T03:33:12.823 INFO:teuthology.orchestra.run.smithi122.stdout:(38/38): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 10 MB/s | 120 MB 00:11 2024-09-22T03:33:12.827 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:33:12.828 INFO:teuthology.orchestra.run.smithi122.stdout:Total 11 MB/s | 134 MB 00:12 2024-09-22T03:33:13.012 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:33:13.055 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:33:13.055 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:33:14.131 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:33:14.131 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:33:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:33:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-09-22T03:33:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-09-22T03:33:16.244 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-09-22T03:33:16.315 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-09-22T03:33:16.360 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-09-22T03:33:16.393 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-09-22T03:33:16.485 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-09-22T03:33:16.572 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-09-22T03:33:16.612 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/38 2024-09-22T03:33:16.664 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-09-22T03:33:16.694 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-09-22T03:33:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-09-22T03:33:16.762 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-09-22T03:33:16.802 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-09-22T03:33:16.839 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-09-22T03:33:16.873 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-09-22T03:33:16.921 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-09-22T03:33:16.954 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-09-22T03:33:17.026 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-09-22T03:33:17.166 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-09-22T03:33:17.217 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-09-22T03:33:36.034 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-09-22T03:33:36.073 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:33:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:33:36.102 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:33:36.102 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:33:36.102 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:33:36.102 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:33:36.102 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:36.206 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-09-22T03:33:36.253 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-09-22T03:33:36.301 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-09-22T03:33:36.357 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-09-22T03:33:36.544 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-09-22T03:33:36.599 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-09-22T03:33:36.668 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-09-22T03:33:36.758 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-09-22T03:33:36.790 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-09-22T03:33:36.914 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-09-22T03:33:37.081 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 34/38 2024-09-22T03:33:37.117 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:33:37.140 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:33:37.141 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:33:37.141 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:33:37.141 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:33:37.141 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:33:37.141 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:33:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:33:37.625 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:33:37.625 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:33:37.625 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:33:37.625 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:33:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:37.843 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:33:37.867 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:33:37.867 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:33:37.867 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:33:37.867 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:33:37.867 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:33:37.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:33:49.173 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:33:49.173 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/38 2024-09-22T03:33:49.173 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-09-22T03:33:49.174 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-09-22T03:33:49.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-09-22T03:33:49.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-09-22T03:33:49.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-09-22T03:33:49.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-09-22T03:33:49.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-09-22T03:33:49.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-09-22T03:33:49.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-09-22T03:33:49.175 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-09-22T03:33:49.176 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-09-22T03:33:49.177 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-09-22T03:33:49.177 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-09-22T03:33:49.177 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-09-22T03:33:49.177 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-09-22T03:33:49.177 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-09-22T03:33:49.177 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-09-22T03:33:49.177 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-09-22T03:33:49.177 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 36/38 2024-09-22T03:33:49.178 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-09-22T03:33:49.969 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:33:49.970 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:33:49.971 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:33:49.972 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:33:49.973 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:49.973 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:33:50.244 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-09-22T03:33:50.848 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:56 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:33:50.941 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:33:50.984 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:33:50.985 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:33:50.986 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:33:51.056 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-09-22T03:33:51.634 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:57 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:33:51.765 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:51.766 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 765 k 2024-09-22T03:33:51.767 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 771 k 2024-09-22T03:33:51.767 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:33:52.137 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.0 MB/s | 765 kB 00:00 2024-09-22T03:33:52.137 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:33:52.138 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.0 MB/s | 765 kB 00:00 2024-09-22T03:33:52.138 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:33:52.143 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:33:52.143 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:33:52.149 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:33:52.149 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:33:52.219 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:33:52.697 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:33:52.791 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:33:53.174 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:33:53.410 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:33:53.411 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:53.411 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:33:53.411 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:33:53.411 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:53.411 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:33:53.622 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:33:54.212 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:00 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:33:54.346 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:33:54.347 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:54.347 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-22T03:33:54.347 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:54.347 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 147 k 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 147 k 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 481 k 2024-09-22T03:33:54.348 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:33:54.559 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 698 kB/s | 147 kB 00:00 2024-09-22T03:33:54.560 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:33:54.560 INFO:teuthology.orchestra.run.smithi122.stdout:Total 691 kB/s | 147 kB 00:00 2024-09-22T03:33:54.561 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:33:54.568 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:33:54.568 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:33:54.625 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:33:54.626 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:33:54.773 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:33:54.844 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:33:54.869 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:33:54.869 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:33:54.869 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:33:54.869 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:33:55.796 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:55.797 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:33:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:55.797 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:55.797 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:33:56.020 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-09-22T03:33:56.614 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:02 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:33:56.703 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:33:56.745 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:33:56.746 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:33:56.746 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:33:56.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:33:57.378 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:03 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:33:57.509 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:33:57.510 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:57.510 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:33:57.510 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:33:57.511 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:33:57.512 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:33:57.512 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:33:57.512 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:57.512 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:33:57.512 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:33:57.512 INFO:teuthology.orchestra.run.smithi122.stdout:Install 11 Packages 2024-09-22T03:33:57.512 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:33:57.512 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 7.7 M 2024-09-22T03:33:57.513 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 101 M 2024-09-22T03:33:57.513 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:33:58.155 INFO:teuthology.orchestra.run.smithi122.stdout:(1/11): ceph-grafana-dashboards-19.1.1-373.g4e4 216 kB/s | 32 kB 00:00 2024-09-22T03:33:58.180 INFO:teuthology.orchestra.run.smithi122.stdout:(2/11): ceph-prometheus-alerts-19.1.1-373.g4e45 95 kB/s | 16 kB 00:00 2024-09-22T03:33:58.398 INFO:teuthology.orchestra.run.smithi122.stdout:(3/11): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 9.2 MB/s | 3.6 MB 00:00 2024-09-22T03:33:58.423 INFO:teuthology.orchestra.run.smithi122.stdout:(4/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-09-22T03:33:58.473 INFO:teuthology.orchestra.run.smithi122.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-09-22T03:33:58.515 INFO:teuthology.orchestra.run.smithi122.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 210 kB/s | 19 kB 00:00 2024-09-22T03:33:58.540 INFO:teuthology.orchestra.run.smithi122.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 6.1 MB/s | 144 kB 00:00 2024-09-22T03:33:58.566 INFO:teuthology.orchestra.run.smithi122.stdout:(8/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.2 MB/s | 548 kB 00:00 2024-09-22T03:33:58.591 INFO:teuthology.orchestra.run.smithi122.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 611 kB/s | 31 kB 00:00 2024-09-22T03:33:58.616 INFO:teuthology.orchestra.run.smithi122.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-22T03:33:58.683 INFO:teuthology.orchestra.run.smithi122.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-09-22T03:33:58.687 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:33:58.687 INFO:teuthology.orchestra.run.smithi122.stdout:Total 6.5 MB/s | 7.7 MB 00:01 2024-09-22T03:33:58.717 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:33:58.733 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:33:58.734 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:33:58.947 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:33:58.947 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:33:59.440 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:33:59.514 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:33:59.545 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:33:59.664 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:33:59.786 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:33:59.840 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:33:59.923 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:33:59.983 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:34:00.012 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:34:00.051 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 9/11 2024-09-22T03:34:01.111 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 10/11 2024-09-22T03:34:01.153 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:34:02.464 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:34:02.465 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:34:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:34:02.785 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:02.785 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:34:02.786 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:02.787 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:03.013 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:34:03.592 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:09 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:03.725 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:34:03.726 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-09-22T03:34:03.727 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:03.728 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 38 M 2024-09-22T03:34:03.728 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 212 M 2024-09-22T03:34:03.728 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:04.982 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 205 kB/s | 191 kB 00:00 2024-09-22T03:34:05.058 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 796 kB/s | 801 kB 00:01 2024-09-22T03:34:05.116 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-22T03:34:05.199 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 219 kB/s | 18 kB 00:00 2024-09-22T03:34:05.292 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 420 kB/s | 38 kB 00:00 2024-09-22T03:34:05.484 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-09-22T03:34:05.542 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-22T03:34:05.684 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-09-22T03:34:05.768 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 10 MB/s | 4.9 MB 00:00 2024-09-22T03:34:05.793 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.3 MB/s | 250 kB 00:00 2024-09-22T03:34:06.110 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-22T03:34:07.760 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): ceph-mgr-diskprediction-local-19.1.1-3 2.0 MB/s | 7.4 MB 00:03 2024-09-22T03:34:07.761 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:07.761 INFO:teuthology.orchestra.run.smithi122.stdout:Total 9.5 MB/s | 38 MB 00:04 2024-09-22T03:34:08.121 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:08.148 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:08.148 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:08.569 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:08.569 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:09.328 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:09.386 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:34:09.425 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:34:09.527 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:34:09.576 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:34:09.760 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:34:09.792 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:34:09.868 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:34:10.541 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:34:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:34:11.914 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:34:12.941 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:34:12.975 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:34:15.570 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:34:15.570 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-22T03:34:15.570 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:34:15.571 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:34:16.036 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:34:16.037 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:34:16.037 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:34:16.037 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:34:16.037 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:16.037 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:16.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:34:17.015 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:23 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:17.147 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:17.148 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:17.148 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:34:17.148 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:34:17.149 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:34:17.150 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:34:17.150 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:17.150 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:17.150 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:17.150 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-09-22T03:34:17.150 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:17.150 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.2 M 2024-09-22T03:34:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-09-22T03:34:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:17.601 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 293 kB/s | 49 kB 00:00 2024-09-22T03:34:17.717 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 90 kB/s | 26 kB 00:00 2024-09-22T03:34:17.743 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 62 kB/s | 19 kB 00:00 2024-09-22T03:34:18.085 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 432 kB/s | 159 kB 00:00 2024-09-22T03:34:18.152 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 403 kB/s | 222 kB 00:00 2024-09-22T03:34:18.194 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 494 kB/s | 54 kB 00:00 2024-09-22T03:34:18.219 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-22T03:34:18.261 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 538 kB/s | 279 kB 00:00 2024-09-22T03:34:18.295 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-22T03:34:18.320 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-22T03:34:18.345 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-22T03:34:18.395 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-22T03:34:18.504 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-google-auth-2.34.0-1.el9.noarc 793 kB/s | 220 kB 00:00 2024-09-22T03:34:18.506 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:18.506 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.6 MB/s | 2.2 MB 00:01 2024-09-22T03:34:18.576 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:18.592 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:18.592 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:18.680 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:18.680 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:19.187 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:19.348 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:34:19.421 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:34:19.486 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:34:19.523 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:34:19.564 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:34:19.674 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:34:19.784 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:34:19.833 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:34:20.540 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:34:20.589 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:34:20.622 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:34:20.678 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:34:20.709 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:34:21.480 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:34:21.480 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:34:21.481 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:34:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:34:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:21.819 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:34:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:34:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:34:21.819 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:21.820 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:22.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:34:22.679 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:28 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:22.812 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:22.812 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:34:22.813 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:22.814 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:22.814 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:22.814 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-09-22T03:34:22.814 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:22.814 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 7.0 M 2024-09-22T03:34:22.815 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 33 M 2024-09-22T03:34:22.815 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:23.649 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 684 kB/s | 146 kB 00:00 2024-09-22T03:34:23.775 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.8 MB/s | 548 kB 00:00 2024-09-22T03:34:23.800 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-22T03:34:23.833 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-22T03:34:24.200 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 325 kB/s | 248 kB 00:00 2024-09-22T03:34:24.567 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.3 MB/s | 6.0 MB 00:01 2024-09-22T03:34:24.570 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:24.570 INFO:teuthology.orchestra.run.smithi122.stdout:Total 4.0 MB/s | 7.0 MB 00:01 2024-09-22T03:34:24.643 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:24.654 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:24.654 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:24.755 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:24.755 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:25.060 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:25.243 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:34:25.311 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:34:25.847 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:34:25.947 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:34:26.010 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:34:26.040 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:34:26.821 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:34:26.821 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-22T03:34:26.821 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:34:26.821 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:34:26.821 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:34:26.821 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:34:27.100 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:27.357 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-09-22T03:34:27.934 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:33 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:28.068 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:28.068 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:28.068 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:34:28.068 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:28.068 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:28.068 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 868 k 2024-09-22T03:34:28.068 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-09-22T03:34:28.068 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:34:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:28.069 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:28.069 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:28.069 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-09-22T03:34:28.069 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:28.069 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 948 k 2024-09-22T03:34:28.069 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 3.1 M 2024-09-22T03:34:28.069 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:28.628 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 3.0 MB/s | 868 kB 00:00 2024-09-22T03:34:28.978 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 126 kB/s | 80 kB 00:00 2024-09-22T03:34:28.978 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:28.979 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.0 MB/s | 948 kB 00:00 2024-09-22T03:34:28.991 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:29.015 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:29.015 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:29.096 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:29.096 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:29.287 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:29.430 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:34:29.502 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:34:30.198 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:34:30.198 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-22T03:34:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:34:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:30.524 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:34:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:30.524 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:30.783 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-volume 2024-09-22T03:34:31.376 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:37 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:31.521 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:31.522 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:31.522 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-09-22T03:34:31.522 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:31.522 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:31.522 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-22T03:34:31.522 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:31.522 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:31.523 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:31.523 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-22T03:34:31.523 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:31.523 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 285 k 2024-09-22T03:34:31.523 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.3 M 2024-09-22T03:34:31.523 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:31.767 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-22T03:34:31.767 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:31.767 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-22T03:34:31.768 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:31.776 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:31.776 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:31.803 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:31.803 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:32.029 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:32.144 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:34:32.161 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:34:32.161 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:34:32.161 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:34:32.161 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:33.146 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:34:33.147 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:33.147 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:33.147 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:33.147 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:33.147 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:33.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-09-22T03:34:34.023 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:40 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:34.158 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:34.159 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:34.159 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-22T03:34:34.159 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 127 k 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 457 k 2024-09-22T03:34:34.160 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:34.364 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 622 kB/s | 127 kB 00:00 2024-09-22T03:34:34.365 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:34.365 INFO:teuthology.orchestra.run.smithi122.stdout:Total 617 kB/s | 127 kB 00:00 2024-09-22T03:34:34.365 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:34.371 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:34.371 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:34.435 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:34.435 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:34.617 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:34.729 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:35.457 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:35.457 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:35.457 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:35.457 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:35.457 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:35.457 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:35.666 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-09-22T03:34:36.261 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:42 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:36.352 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:36.397 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:36.398 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:34:36.398 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:36.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-09-22T03:34:37.059 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:43 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:37.191 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:37.191 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:37.192 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 32 k 2024-09-22T03:34:37.193 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 142 k 2024-09-22T03:34:37.193 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:37.343 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 210 kB/s | 32 kB 00:00 2024-09-22T03:34:37.343 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:37.344 INFO:teuthology.orchestra.run.smithi122.stdout:Total 209 kB/s | 32 kB 00:00 2024-09-22T03:34:37.344 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:37.347 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:37.347 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:37.368 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:37.368 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:38.206 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:38.206 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:38.206 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:38.206 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:38.206 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:38.207 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:38.383 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-09-22T03:34:38.965 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:44 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:39.054 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:39.097 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:39.099 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:34:39.099 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:39.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-09-22T03:34:39.738 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:45 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:39.829 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:39.873 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:39.874 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:34:39.874 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:39.934 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-09-22T03:34:40.506 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:46 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:40.594 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:40.637 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:40.638 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:34:40.638 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:40.699 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-09-22T03:34:41.272 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:47 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:41.362 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:41.405 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:41.406 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:34:41.406 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:41.470 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-09-22T03:34:42.047 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:48 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:42.135 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:42.179 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:42.180 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:34:42.180 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:42.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-09-22T03:34:42.835 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:48 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:42.924 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:42.968 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:42.969 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-09-22T03:34:42.969 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:43.037 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-09-22T03:34:43.612 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:49 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:43.746 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-22T03:34:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 85 k 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 256 k 2024-09-22T03:34:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 190 kB/s | 85 kB 00:00 2024-09-22T03:34:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Total 190 kB/s | 85 kB 00:00 2024-09-22T03:34:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:44.268 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:44.268 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:44.420 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:44.528 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:44.914 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:45.197 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:45.198 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:45.198 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:45.198 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:45.198 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:45.198 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:45.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-09-22T03:34:46.037 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:52 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:46.172 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:46.173 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-22T03:34:46.174 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:46.174 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.1 M 2024-09-22T03:34:46.174 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 14 M 2024-09-22T03:34:46.174 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:47.034 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 3.5 MB/s | 3.1 MB 00:00 2024-09-22T03:34:47.035 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:47.035 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.5 MB/s | 3.1 MB 00:00 2024-09-22T03:34:47.035 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:47.044 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:47.044 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:47.125 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:47.125 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:47.579 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:47.643 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:47.667 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:47.667 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:34:47.667 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:34:47.667 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:34:47.667 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:34:47.667 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:48.837 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:48.838 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:48.838 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:48.838 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:48.838 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:48.838 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:49.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-09-22T03:34:49.656 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:55 ago on Sun 22 Sep 2024 03:31:54 AM UTC. 2024-09-22T03:34:49.790 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-09-22T03:34:49.790 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 169 k 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 169 k 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 540 k 2024-09-22T03:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-09-22T03:34:50.021 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 739 kB/s | 169 kB 00:00 2024-09-22T03:34:50.022 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:50.022 INFO:teuthology.orchestra.run.smithi122.stdout:Total 734 kB/s | 169 kB 00:00 2024-09-22T03:34:50.022 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-09-22T03:34:50.029 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-09-22T03:34:50.029 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-09-22T03:34:50.093 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-09-22T03:34:50.093 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-09-22T03:34:50.244 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-09-22T03:34:50.360 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:50.805 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:51.058 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:34:51.058 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:51.058 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-09-22T03:34:51.058 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:51.058 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:51.058 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-09-22T03:34:51.329 DEBUG:teuthology.parallel:result is None 2024-09-22T03:34:51.329 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:34:51.474 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:34:51.496 INFO:teuthology.orchestra.run.smithi122.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-22T03:34:51.496 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-22T03:34:51.497 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-22T03:34:51.498 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T03:34:51.498 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:51.498 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:34:51.563 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T03:34:51.564 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:51.564 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:34:51.628 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:34:51.692 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T03:34:51.693 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:51.693 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:34:51.757 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:34:51.821 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T03:34:51.822 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:51.822 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:34:51.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:34:51.949 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T03:34:52.021 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T03:34:52.021 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-09-22T03:34:52.068 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T03:34:52.068 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:34:52.092 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T03:34:52.092 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:34:52.164 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T03:34:52.164 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T03:34:52.165 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914779', 'branch': 'squid', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_omap_write}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914779', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'squid', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1158, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGJYukSc47m/qSHkVP+WSf278IHS1zf2QuK7xT9+NhTNU5d1nIt/KgAYwNztTICj6V80R1CcJlJ/Th2u8RrF+U='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'cluster': 'ceph'}}, {'radosbench': {'clients': ['client.0'], 'objectsize': 4096, 'size': 4096, 'time': 300, 'write-omap': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T03:34:52.166 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:34:52.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:34:52.277 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:52.277 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:34:52.333 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:34:52.333 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-09-22T03:34:52.388 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:34:52.388 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:34:52.388 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-22T03:34:52.388 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:34:52.389 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:34:52.389 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-22 03:34:47.915398299 +0000 2024-09-22T03:34:52.389 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-22 03:31:17.284925371 +0000 2024-09-22T03:34:52.389 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-22 03:31:17.284925371 +0000 2024-09-22T03:34:52.389 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-22 03:31:17.284925371 +0000 2024-09-22T03:34:52.389 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:34:52.451 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-22T03:34:52.451 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-22T03:34:52.451 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000200587 s, 2.6 MB/s 2024-09-22T03:34:52.452 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:34:52.508 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-22 03:34:47.915398299 +0000 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-22 03:31:17.609918084 +0000 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-22 03:31:17.609918084 +0000 2024-09-22T03:34:52.563 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-22 03:31:17.609918084 +0000 2024-09-22T03:34:52.564 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:34:52.626 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-22T03:34:52.627 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-22T03:34:52.627 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000195244 s, 2.6 MB/s 2024-09-22T03:34:52.628 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:34:52.684 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-09-22T03:34:52.739 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:34:52.739 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:34:52.739 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-22T03:34:52.739 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:34:52.739 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:34:52.739 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-22 03:34:47.915398299 +0000 2024-09-22T03:34:52.739 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-22 03:31:17.925910997 +0000 2024-09-22T03:34:52.740 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-22 03:31:17.925910997 +0000 2024-09-22T03:34:52.740 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-22 03:31:17.925910997 +0000 2024-09-22T03:34:52.740 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:34:52.802 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-22T03:34:52.802 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-22T03:34:52.802 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000216157 s, 2.4 MB/s 2024-09-22T03:34:52.803 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:34:52.860 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-09-22T03:34:52.915 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:34:52.915 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:34:52.915 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-22T03:34:52.915 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:34:52.915 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:34:52.915 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-09-22 03:34:47.916398278 +0000 2024-09-22T03:34:52.916 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-09-22 03:31:18.237904001 +0000 2024-09-22T03:34:52.916 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-09-22 03:31:18.237904001 +0000 2024-09-22T03:34:52.916 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-09-22 03:31:18.237904001 +0000 2024-09-22T03:34:52.916 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:34:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-09-22T03:34:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-09-22T03:34:52.978 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000184898 s, 2.8 MB/s 2024-09-22T03:34:52.979 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:34:53.035 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:34:53.035 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:34:53.035 INFO:tasks.ceph:Generating config... 2024-09-22T03:34:53.036 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T03:34:53.036 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T03:34:53.036 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T03:34:53.036 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-22T03:34:53.036 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-22T03:34:53.036 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-22T03:34:53.036 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-22T03:34:53.037 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-22T03:34:53.037 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] auth client required = none 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] auth service required = none 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] auth supported = none 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-22T03:34:53.037 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-22T03:34:53.038 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-22T03:34:53.039 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-22T03:34:53.040 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T03:34:53.041 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T03:34:53.042 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T03:34:53.042 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T03:34:53.042 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T03:34:53.042 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T03:34:53.042 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T03:34:53.042 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T03:34:53.042 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-22T03:34:53.042 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-22T03:34:53.042 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-22T03:34:53.042 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T03:34:53.042 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-22T03:34:53.042 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-22T03:34:53.042 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-22T03:34:53.042 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-22T03:34:53.043 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T03:34:53.043 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T03:34:53.149 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T03:34:53.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T03:34:53.199 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:34:53.224 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.122')] 2024-09-22T03:34:53.224 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.122', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-22T03:34:53.225 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:53.225 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:34:53.279 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.122 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:34:53.366 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:34:53.365+0000 7fc2cf089e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:34:53.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:34:53.365+0000 7fc2cf089e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:34:53.367 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:34:53.365+0000 7fc2cf089e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:34:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:34:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: generated fsid 238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77 2024-09-22T03:34:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:setting min_mon_release = quincy 2024-09-22T03:34:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:epoch 0 2024-09-22T03:34:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:fsid 238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77 2024-09-22T03:34:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:last_changed 2024-09-22T03:34:53.366661+0000 2024-09-22T03:34:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:created 2024-09-22T03:34:53.366661+0000 2024-09-22T03:34:53.368 INFO:teuthology.orchestra.run.smithi122.stdout:min_mon_release 17 (quincy) 2024-09-22T03:34:53.369 INFO:teuthology.orchestra.run.smithi122.stdout:election_strategy: 1 2024-09-22T03:34:53.369 INFO:teuthology.orchestra.run.smithi122.stdout:0: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.a 2024-09-22T03:34:53.369 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T03:34:53.369 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:34:53.384 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77... 2024-09-22T03:34:53.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:34:53.466 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-09-22T03:34:53.466 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-09-22T03:34:53.466 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:34:53.466 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = none 2024-09-22T03:34:53.466 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.466 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-09-22T03:34:53.466 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.466 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:34:53.467 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.468 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = 172.21.15.122 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: auth client required = none 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: auth cluster required = none 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: auth service required = none 2024-09-22T03:34:53.469 INFO:teuthology.orchestra.run.smithi122.stdout: debug asok = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug bdev = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug buffer = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug civetweb = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug compressor = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug context = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug crush = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug crypto = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug dpdk = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug eventtrace = 0/0 2024-09-22T03:34:53.470 INFO:teuthology.orchestra.run.smithi122.stdout: debug filer = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug filestore = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug finisher = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug fuse = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug heartbeatmap = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug javaclient = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug journal = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug journaler = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug kstore = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug leveldb = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug lockdep = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 0/0 2024-09-22T03:34:53.471 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds locker = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds log = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds log expire = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds migrator = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug memdb = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgrc = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug monc = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug objclass = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug objectcacher = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug objecter = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug optracker = 0/0 2024-09-22T03:34:53.472 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug perfcounter = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug rados = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd mirror = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd replay = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug refs = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug rgw = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug rgw lifecycle = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug rgw sync = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug striper = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug throttle = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug timer = 0/0 2024-09-22T03:34:53.473 INFO:teuthology.orchestra.run.smithi122.stdout: debug tp = 0/0 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: ms client mode = crc 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: ms cluster mode = crc 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: ms mon client mode = crc 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: ms mon cluster mode = crc 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: ms mon service mode = crc 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: ms service mode = crc 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-09-22T03:34:53.474 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.475 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: debug monc = 20 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: crimson osd obc lru size = 10 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: debug filestore = 0/0 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: debug journal = 0/0 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 0/0 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 0/0 2024-09-22T03:34:53.476 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = seastore 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: osd shutdown pgref assert = True 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 0/0 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 0/0 2024-09-22T03:34:53.477 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 0/0 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:34:53.478 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default crimson = True 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:34:53.479 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-09-22T03:34:53.480 INFO:teuthology.orchestra.run.smithi122.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:34:53.480 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-09-22T03:34:53.480 INFO:teuthology.orchestra.run.smithi122.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:34:53.480 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.3] 2024-09-22T03:34:53.480 INFO:teuthology.orchestra.run.smithi122.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:34:53.481 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T03:34:53.481 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T03:34:53.581 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T03:34:53.581 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:53.581 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T03:34:53.596 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:53.596 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T03:34:53.651 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:34:53.651 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:53.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:34:53.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:34:53.725 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:34:53.725 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:34:53.780 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T03:34:53.780 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T03:34:53.781 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:34:53.876 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:34:53.879 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T03:34:53.879 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T03:34:53.879 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T03:34:53.925 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T03:34:53.938 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T03:34:53.938 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:34:53.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T03:34:53.963 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:34:53.963 INFO:tasks.ceph:role: osd.0 2024-09-22T03:34:53.963 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:34:53.963 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:34:54.035 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:34:54.035 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:34:54.035 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:34:54.035 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:34:54.035 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:34:54.036 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:34:54.036 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:34:54.036 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:34:54.036 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:34:54.036 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:34:54.509 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-22T03:34:54.511 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-22T03:34:54.511 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T03:34:54.649 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T03:34:54.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T03:34:54.749 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:34:54.749 INFO:tasks.ceph:role: osd.1 2024-09-22T03:34:54.749 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:34:54.749 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:34:54.821 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:34:54.821 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:34:54.822 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:34:54.822 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:34:54.822 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:34:54.822 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:34:54.822 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:34:54.822 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:34:54.822 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:34:54.822 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:34:55.288 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-22T03:34:55.290 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-22T03:34:55.290 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T03:34:55.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T03:34:55.422 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T03:34:55.489 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:34:55.489 INFO:tasks.ceph:role: osd.2 2024-09-22T03:34:55.489 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:34:55.489 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:34:55.563 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:34:56.044 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-09-22T03:34:56.047 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-09-22T03:34:56.047 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-22T03:34:56.191 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T03:34:56.227 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:34:56.802 INFO:teuthology.orchestra.run.smithi122.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:34:56.803 INFO:teuthology.orchestra.run.smithi122.stderr:INFO 2024-09-22 03:34:56,803 seastar - Reactor backend: linux-aio 2024-09-22T03:34:56.825 INFO:teuthology.orchestra.run.smithi122.stderr:==27600==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:34:56.880 INFO:teuthology.orchestra.run.smithi122.stderr:DEBUG 2024-09-22 03:34:56,879 [shard 0:main] seastar - reactor::drain 2024-09-22T03:34:57.374 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:34:57.374 INFO:teuthology.orchestra.run.smithi122.stderr:Suppressions used: 2024-09-22T03:34:57.374 INFO:teuthology.orchestra.run.smithi122.stderr: count bytes template 2024-09-22T03:34:57.374 INFO:teuthology.orchestra.run.smithi122.stderr: 1 8 MallocExtension::Register 2024-09-22T03:34:57.374 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:34:57.375 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-22T03:34:57.427 INFO:teuthology.orchestra.run.smithi122.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:34:57.429 INFO:teuthology.orchestra.run.smithi122.stderr:INFO 2024-09-22 03:34:57,428 seastar - Reactor backend: linux-aio 2024-09-22T03:34:57.464 INFO:teuthology.orchestra.run.smithi122.stderr:==27598==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:34:57.613 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x943d7 0x94d46 0x2ff76305 0x3014c053 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:34:57.614 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:34:57.614 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xd591f 0xd5ac0 0xd554b 0x287b7 0xb4674 0x2b8abfc6 0x2bd7eef6 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:34:57.614 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:34:57.908 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2ff76234 0x2ff7730f 0x2ff775b5 0x3014ba39 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:34:57.908 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x3005227f 0x30145156 0x30145a3b 0x3014b444 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:34:57.908 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:34:57.908 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:34:58.658 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:34:58.658 INFO:teuthology.orchestra.run.smithi122.stderr:Suppressions used: 2024-09-22T03:34:58.658 INFO:teuthology.orchestra.run.smithi122.stderr: count bytes template 2024-09-22T03:34:58.659 INFO:teuthology.orchestra.run.smithi122.stderr: 1 8 MallocExtension::Register 2024-09-22T03:34:58.659 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:34:58.659 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-22T03:34:58.659 INFO:teuthology.orchestra.run.smithi122.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77 2024-09-22T03:34:58.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T03:34:58.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:34:59.280 INFO:teuthology.orchestra.run.smithi122.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:34:59.281 INFO:teuthology.orchestra.run.smithi122.stderr:INFO 2024-09-22 03:34:59,281 seastar - Reactor backend: linux-aio 2024-09-22T03:34:59.299 INFO:teuthology.orchestra.run.smithi122.stderr:==27649==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:34:59.346 INFO:teuthology.orchestra.run.smithi122.stderr:DEBUG 2024-09-22 03:34:59,346 [shard 0:main] seastar - reactor::drain 2024-09-22T03:34:59.831 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:34:59.831 INFO:teuthology.orchestra.run.smithi122.stderr:Suppressions used: 2024-09-22T03:34:59.831 INFO:teuthology.orchestra.run.smithi122.stderr: count bytes template 2024-09-22T03:34:59.831 INFO:teuthology.orchestra.run.smithi122.stderr: 1 8 MallocExtension::Register 2024-09-22T03:34:59.831 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:34:59.831 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-22T03:34:59.883 INFO:teuthology.orchestra.run.smithi122.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:34:59.884 INFO:teuthology.orchestra.run.smithi122.stderr:INFO 2024-09-22 03:34:59,884 seastar - Reactor backend: linux-aio 2024-09-22T03:34:59.921 INFO:teuthology.orchestra.run.smithi122.stderr:==27647==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:00.072 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x39761 0x94456 0x94d46 0x2ff35bd4 0x2ffc28cc 0x3014b686 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:00.073 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:35:00.073 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2ff7657f 0x2ff7730f 0x2ff775b5 0x3014ba39 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:00.073 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:35:00.376 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c605 0x2857c 0xb6323 0x2fcc5109 0x2fcc5a6e 0x2ffbd164 0x3014d7dd 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:00.376 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x3014b914 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd7c6de 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:00.376 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:35:00.376 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb2a36eee 0xffffffffb25f61bf 0xffffffffb25f64c0 0xffffffffb24908cb 0xffffffffb27a8199 0xffffffffb278e203 0xffffffffb279699b 0xffffffffb2796e1c 0xffffffffb248c884 0xffffffffb30d1db2 0xffffffffb3200bb2 2024-09-22T03:35:00.433 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b2d 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:35:00.434 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:35:01.105 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:35:01.105 INFO:teuthology.orchestra.run.smithi122.stderr:Suppressions used: 2024-09-22T03:35:01.105 INFO:teuthology.orchestra.run.smithi122.stderr: count bytes template 2024-09-22T03:35:01.105 INFO:teuthology.orchestra.run.smithi122.stderr: 1 8 MallocExtension::Register 2024-09-22T03:35:01.106 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:35:01.106 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-22T03:35:01.106 INFO:teuthology.orchestra.run.smithi122.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77 2024-09-22T03:35:01.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T03:35:01.176 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:35:01.719 INFO:teuthology.orchestra.run.smithi122.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:35:01.720 INFO:teuthology.orchestra.run.smithi122.stderr:INFO 2024-09-22 03:35:01,720 seastar - Reactor backend: linux-aio 2024-09-22T03:35:01.742 INFO:teuthology.orchestra.run.smithi122.stderr:==27696==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:01.788 INFO:teuthology.orchestra.run.smithi122.stderr:DEBUG 2024-09-22 03:35:01,788 [shard 0:main] seastar - reactor::drain 2024-09-22T03:35:02.273 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:35:02.273 INFO:teuthology.orchestra.run.smithi122.stderr:Suppressions used: 2024-09-22T03:35:02.273 INFO:teuthology.orchestra.run.smithi122.stderr: count bytes template 2024-09-22T03:35:02.273 INFO:teuthology.orchestra.run.smithi122.stderr: 1 8 MallocExtension::Register 2024-09-22T03:35:02.273 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:35:02.273 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-22T03:35:02.329 INFO:teuthology.orchestra.run.smithi122.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:35:02.330 INFO:teuthology.orchestra.run.smithi122.stderr:INFO 2024-09-22 03:35:02,330 seastar - Reactor backend: linux-aio 2024-09-22T03:35:02.358 INFO:teuthology.orchestra.run.smithi122.stderr:==27694==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:02.507 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c5f5 0x282da 0xb49e1 0x2b8ab36f 0x2bd807cb 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:02.508 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x3014b9bf 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:02.508 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:35:02.508 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:35:02.815 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c1d0 0x2c5a7 0x2857c 0xb6323 0x2ffbca31 0x3014d7dd 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd7c6de 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:02.815 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:35:02.815 INFO:teuthology.orchestra.run.smithi122.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c5c1 0x2857c 0xb6323 0x2fd9d660 0x2bd7b9f1 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:02.815 INFO:teuthology.orchestra.run.smithi122.stderr:kernel callstack: 2024-09-22T03:35:03.559 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:35:03.559 INFO:teuthology.orchestra.run.smithi122.stderr:Suppressions used: 2024-09-22T03:35:03.559 INFO:teuthology.orchestra.run.smithi122.stderr: count bytes template 2024-09-22T03:35:03.559 INFO:teuthology.orchestra.run.smithi122.stderr: 1 8 MallocExtension::Register 2024-09-22T03:35:03.559 INFO:teuthology.orchestra.run.smithi122.stderr:----------------------------------------------------- 2024-09-22T03:35:03.559 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-22T03:35:03.560 INFO:teuthology.orchestra.run.smithi122.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77 2024-09-22T03:35:03.606 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T03:35:03.631 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T03:35:03.631 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:35:03.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T03:35:03.695 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:35:03.695 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T03:35:03.760 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:35:03.760 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T03:35:03.824 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:35:03.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T03:35:03.888 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:35:03.888 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T03:35:03.943 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T03:35:03.943 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBdkO9mXO43NBAAL644YNauDhe2Wk/gLXMWJg== 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBhkO9m4jTyHhAADtrzY+/F5Cei2MEItNwtmA== 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBjkO9mGy0fOhAADXaac64Kv8ycrD5ZkkDsog== 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBmkO9mDoWoGBAAhWkO9O/EkCwYuEw5QTOFPg== 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-09-22T03:35:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBdkO9mRJUnNxAAO3T7JkfZ1t/SQTD6K+xxCw== 2024-09-22T03:35:04.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T03:35:04.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:35:04.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:35:04.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:35:04.270 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:35:04.329 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T03:35:04.330 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T03:35:04.354 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T03:35:04.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:04.450+0000 7fac4ea16b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:04.450+0000 7fac4ea16b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.465 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T03:35:04.489 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:35:04.544 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T03:35:04.544 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T03:35:04.544 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T03:35:04.587 INFO:tasks.ceph.mon.a:Started 2024-09-22T03:35:04.587 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T03:35:04.587 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T03:35:04.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T03:35:04.589 INFO:tasks.ceph.mgr.x:Started 2024-09-22T03:35:04.589 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T03:35:04.589 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T03:35:04.664 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-09-22T03:35:04.663+0000 7f40d5c0eb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.664 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-09-22T03:35:04.663+0000 7f40d5c0eb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.664 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-09-22T03:35:04.663+0000 7f40d5c0eb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.665 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:04.665+0000 7f6ed4524f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.694 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:04.688+0000 7f502d46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.696 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:04.696+0000 7f502d46c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.762 INFO:teuthology.orchestra.run.smithi122.stderr:adjusted tunables profile to default 2024-09-22T03:35:04.774 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T03:35:04.774 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T03:35:04.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T03:35:04.869 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:04.868+0000 7f6ed4524f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.869 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:04.868+0000 7f6ed4524f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.875 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:04.871+0000 7f18f126c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.877 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:04.877+0000 7f18f126c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:04.954 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-09-22T03:35:04.953+0000 7f40cdbf1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T03:35:04.954 INFO:tasks.ceph.mon.a.smithi122.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T03:35:04.954 INFO:tasks.ceph.mon.a.smithi122.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T03:35:04.954 INFO:tasks.ceph.mon.a.smithi122.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T03:35:04.954 INFO:tasks.ceph.mon.a.smithi122.stderr:feature with important data. 2024-09-22T03:35:04.955 INFO:tasks.ceph.mon.a.smithi122.stderr: 2024-09-22T03:35:04.969 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:04.968+0000 7f6ed4524f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-22T03:35:05.009 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:05.009+0000 7f6ed4524f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-22T03:35:05.141 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:05.140+0000 7f6ed4524f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-22T03:35:05.192 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:05.191+0000 7f6ed4524f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-22T03:35:05.357 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:05.356+0000 7f6ed4524f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:35:05.468 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:05.467+0000 7f6ed4524f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-22T03:35:05.774 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T03:35:05.774 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:35:05.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T03:35:05.799 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:35:05.799 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T03:35:05.807 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:05.806+0000 7f6ed4524f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-22T03:35:05.830 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-09-22T03:35:05.830 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T03:35:05.864 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:05.863+0000 7f6ed4524f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-22T03:35:05.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 3a0142b1-d7e8-40f9-a6f7-b9df3a1a1781 0 2024-09-22T03:35:06.021 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:06.020+0000 7f6ed4524f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-22T03:35:06.031 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:06.024+0000 7f021004f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:06.033 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:06.031+0000 7f021004f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:06.106 INFO:teuthology.orchestra.run.smithi122.stdout:0 2024-09-22T03:35:06.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 31b3ebc1-9f0d-4f09-b0ef-b9c53c3ef7af 1 2024-09-22T03:35:06.220 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:06.213+0000 7fcdc4f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:06.222 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:06.221+0000 7fcdc4f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:06.297 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-09-22T03:35:06.308 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 72f1f391-3573-406d-97ad-61cbbc5c1355 2 2024-09-22T03:35:06.412 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:06.405+0000 7f4a28157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:06.414 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:06.413+0000 7f4a28157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:06.487 INFO:teuthology.orchestra.run.smithi122.stdout:2 2024-09-22T03:35:06.498 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T03:35:06.527 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:35:06.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:35:06.570 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:35:06.570 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T03:35:06.599 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:35:06.600 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:35:06.642 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:35:06.642 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T03:35:06.674 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:35:06.674 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:35:06.717 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:35:06.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:35:06.840 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:06.835+0000 7ffbf48ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:06.843 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:06.842+0000 7ffbf48ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:06.912 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:35:06.912 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":6,"fsid":"238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77","created":"2024-09-22T03:35:04.688188+0000","modified":"2024-09-22T03:35:06.486277+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:35:06.486277+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3a0142b1-d7e8-40f9-a6f7-b9df3a1a1781","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"31b3ebc1-9f0d-4f09-b0ef-b9c53c3ef7af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72f1f391-3573-406d-97ad-61cbbc5c1355","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:35:06.937 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T03:35:06.937 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T03:35:07.209 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:07.208+0000 7f6ed4524f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-22T03:35:07.238 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:35:07.267 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:07.265+0000 7f6ed4524f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-22T03:35:07.335 INFO:teuthology.misc.health.smithi122.stderr:2024-09-22T03:35:07.330+0000 7faa37c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:07.337 INFO:teuthology.misc.health.smithi122.stderr:2024-09-22T03:35:07.336+0000 7faa37c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:07.403 INFO:teuthology.misc.health.smithi122.stdout: 2024-09-22T03:35:07.403 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":6,"fsid":"238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77","created":"2024-09-22T03:35:04.688188+0000","modified":"2024-09-22T03:35:06.486277+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:35:06.486277+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3a0142b1-d7e8-40f9-a6f7-b9df3a1a1781","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"31b3ebc1-9f0d-4f09-b0ef-b9c53c3ef7af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72f1f391-3573-406d-97ad-61cbbc5c1355","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:35:07.413 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-22T03:35:07.670 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:07.669+0000 7f6ed4524f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-22T03:35:07.763 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:07.762+0000 7f6ed4524f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-22T03:35:07.879 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:07.877+0000 7f6ed4524f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:35:07.929 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:07.928+0000 7f6ed4524f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-22T03:35:08.054 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.050+0000 7f6ed4524f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-22T03:35:08.109 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.108+0000 7f6ed4524f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-22T03:35:08.158 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.156+0000 7f6ed4524f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-22T03:35:08.278 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.276+0000 7f6ed4524f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-22T03:35:08.329 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.328+0000 7f6ed4524f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-22T03:35:08.382 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.381+0000 7f6ed4524f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-22T03:35:08.452 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.451+0000 7f6ed4524f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-22T03:35:08.499 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.497+0000 7f6ed4524f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-22T03:35:08.545 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:08.544+0000 7f6ed4524f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-22T03:35:09.470 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:09.468+0000 7f6ed4524f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-22T03:35:10.494 INFO:tasks.ceph.mgr.x.smithi122.stderr:2024-09-22T03:35:10.493+0000 7f6e736cb640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T03:35:13.715 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:35:13.820 INFO:teuthology.misc.health.smithi122.stderr:2024-09-22T03:35:13.815+0000 7fa408cfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:13.822 INFO:teuthology.misc.health.smithi122.stderr:2024-09-22T03:35:13.820+0000 7fa408cfa640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:13.996 INFO:teuthology.misc.health.smithi122.stdout: 2024-09-22T03:35:13.996 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77","created":"2024-09-22T03:35:04.688188+0000","modified":"2024-09-22T03:35:13.533118+0000","last_up_change":"2024-09-22T03:35:11.531861+0000","last_in_change":"2024-09-22T03:35:06.486277+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:35:13.522660+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a0142b1-d7e8-40f9-a6f7-b9df3a1a1781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1988525314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":1988525314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1988525314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6825","nonce":1988525314}]},"public_addr":"172.21.15.122:6814/1988525314","cluster_addr":"172.21.15.122:6815/1988525314","heartbeat_back_addr":"172.21.15.122:6826/1988525314","heartbeat_front_addr":"172.21.15.122:6825/1988525314","state":["exists","up"]},{"osd":1,"uuid":"31b3ebc1-9f0d-4f09-b0ef-b9c53c3ef7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":934643915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":934643915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":934643915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":934643915}]},"public_addr":"172.21.15.122:6803/934643915","cluster_addr":"172.21.15.122:6802/934643915","heartbeat_back_addr":"172.21.15.122:6804/934643915","heartbeat_front_addr":"172.21.15.122:6805/934643915","state":["exists","up"]},{"osd":2,"uuid":"72f1f391-3573-406d-97ad-61cbbc5c1355","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":802338473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":802338473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6821","nonce":802338473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":802338473}]},"public_addr":"172.21.15.122:6812/802338473","cluster_addr":"172.21.15.122:6813/802338473","heartbeat_back_addr":"172.21.15.122:6821/802338473","heartbeat_front_addr":"172.21.15.122:6820/802338473","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:35:14.008 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-22T03:35:14.008 INFO:tasks.ceph:Creating RBD pool 2024-09-22T03:35:14.008 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T03:35:14.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:14.103+0000 7fb85e8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:14.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:14.111+0000 7fb85e8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:14.538 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'rbd' created 2024-09-22T03:35:14.558 DEBUG:teuthology.orchestra.run.smithi122:> rbd --cluster ceph pool init rbd 2024-09-22T03:35:14.623 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:14.621+0000 7fdbe5838d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:14.624 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:14.622+0000 7fdbe5838d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:14.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:14.625+0000 7fdbe5838d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:17.592 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T03:35:17.592 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T03:35:17.592 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T03:35:17.592 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T03:35:17.593 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T03:35:17.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:17.696+0000 7f1b88c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:17.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:17.701+0000 7f1b88c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:17.921 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:35:17.931 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3643782545},{"type":"v1","addr":"172.21.15.122:6801","nonce":3643782545}]},"active_addr":"172.21.15.122:6801/3643782545","active_change":"2024-09-22T03:35:09.471820+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":467144556}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":2202649914}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":1600919680}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":153563135}]}]} 2024-09-22T03:35:17.934 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T03:35:17.934 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:35:17.935 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:35:18.038 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.034+0000 7f400f8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.039+0000 7f400f8d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.214 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:35:18.215 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":14,"fsid":"238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77","created":"2024-09-22T03:35:04.688188+0000","modified":"2024-09-22T03:35:17.551685+0000","last_up_change":"2024-09-22T03:35:11.531861+0000","last_in_change":"2024-09-22T03:35:06.486277+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:35:13.522660+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:35:14.300922+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a0142b1-d7e8-40f9-a6f7-b9df3a1a1781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1988525314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":1988525314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1988525314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6825","nonce":1988525314}]},"public_addr":"172.21.15.122:6814/1988525314","cluster_addr":"172.21.15.122:6815/1988525314","heartbeat_back_addr":"172.21.15.122:6826/1988525314","heartbeat_front_addr":"172.21.15.122:6825/1988525314","state":["exists","up"]},{"osd":1,"uuid":"31b3ebc1-9f0d-4f09-b0ef-b9c53c3ef7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":934643915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":934643915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":934643915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":934643915}]},"public_addr":"172.21.15.122:6803/934643915","cluster_addr":"172.21.15.122:6802/934643915","heartbeat_back_addr":"172.21.15.122:6804/934643915","heartbeat_front_addr":"172.21.15.122:6805/934643915","state":["exists","up"]},{"osd":2,"uuid":"72f1f391-3573-406d-97ad-61cbbc5c1355","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":802338473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":802338473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6821","nonce":802338473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":802338473}]},"public_addr":"172.21.15.122:6812/802338473","cluster_addr":"172.21.15.122:6813/802338473","heartbeat_back_addr":"172.21.15.122:6821/802338473","heartbeat_front_addr":"172.21.15.122:6820/802338473","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:35:18.226 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:35:18.227 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:35:18.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.325+0000 7f9fe170e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.331 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.330+0000 7f9fe170e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.502 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:35:18.503 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":14,"fsid":"238d1d4d-4aa4-44f2-adf9-5cd2b2f0fb77","created":"2024-09-22T03:35:04.688188+0000","modified":"2024-09-22T03:35:17.551685+0000","last_up_change":"2024-09-22T03:35:11.531861+0000","last_in_change":"2024-09-22T03:35:06.486277+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:35:13.522660+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:35:14.300922+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a0142b1-d7e8-40f9-a6f7-b9df3a1a1781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":1988525314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6815","nonce":1988525314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1988525314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6825","nonce":1988525314}]},"public_addr":"172.21.15.122:6814/1988525314","cluster_addr":"172.21.15.122:6815/1988525314","heartbeat_back_addr":"172.21.15.122:6826/1988525314","heartbeat_front_addr":"172.21.15.122:6825/1988525314","state":["exists","up"]},{"osd":1,"uuid":"31b3ebc1-9f0d-4f09-b0ef-b9c53c3ef7af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6803","nonce":934643915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":934643915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":934643915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6805","nonce":934643915}]},"public_addr":"172.21.15.122:6803/934643915","cluster_addr":"172.21.15.122:6802/934643915","heartbeat_back_addr":"172.21.15.122:6804/934643915","heartbeat_front_addr":"172.21.15.122:6805/934643915","state":["exists","up"]},{"osd":2,"uuid":"72f1f391-3573-406d-97ad-61cbbc5c1355","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":802338473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6813","nonce":802338473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6821","nonce":802338473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":802338473}]},"public_addr":"172.21.15.122:6812/802338473","cluster_addr":"172.21.15.122:6813/802338473","heartbeat_back_addr":"172.21.15.122:6821/802338473","heartbeat_front_addr":"172.21.15.122:6820/802338473","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:35:18.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:35:18.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:35:18.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:35:18.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.619+0000 7f2f998d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.625 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.619+0000 7fdc5eb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.626 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.620+0000 7f9d278a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.627 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.626+0000 7f9d278a9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.626+0000 7f2f998d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.628 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.626+0000 7fdc5eb97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.654 INFO:teuthology.orchestra.run.smithi122.stdout:38654705671 2024-09-22T03:35:18.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:35:18.655 INFO:teuthology.orchestra.run.smithi122.stdout:34359738376 2024-09-22T03:35:18.655 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:35:18.656 INFO:teuthology.orchestra.run.smithi122.stdout:38654705671 2024-09-22T03:35:18.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:35:18.760 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.755+0000 7f618e397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.761+0000 7f618e397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.805 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.800+0000 7f08fff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.807 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.806+0000 7f08fff97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.819 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.814+0000 7f58dbe6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.822 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:18.820+0000 7f58dbe6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:18.990 INFO:teuthology.orchestra.run.smithi122.stdout:38654705671 2024-09-22T03:35:19.003 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.2 2024-09-22T03:35:19.003 DEBUG:teuthology.parallel:result is None 2024-09-22T03:35:19.008 INFO:teuthology.orchestra.run.smithi122.stdout:38654705671 2024-09-22T03:35:19.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.0 2024-09-22T03:35:19.021 DEBUG:teuthology.parallel:result is None 2024-09-22T03:35:19.043 INFO:teuthology.orchestra.run.smithi122.stdout:34359738376 2024-09-22T03:35:19.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738376 for osd.1 2024-09-22T03:35:19.055 DEBUG:teuthology.parallel:result is None 2024-09-22T03:35:19.055 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:35:19.055 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:35:19.200 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:19.195+0000 7f2fc3c18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:19.202 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:19.200+0000 7f2fc3c18640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:19.373 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:35:19.373 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-22T03:35:19.385 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-09-22T03:35:18.643498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564883+0000","last_change":"2024-09-22T03:35:17.568976+0000","last_active":"2024-09-22T03:35:17.564883+0000","last_peered":"2024-09-22T03:35:17.564883+0000","last_clean":"2024-09-22T03:35:17.564883+0000","last_became_active":"2024-09-22T03:35:15.611852+0000","last_became_peered":"2024-09-22T03:35:15.611852+0000","last_unstale":"2024-09-22T03:35:17.564883+0000","last_undegraded":"2024-09-22T03:35:17.564883+0000","last_fullsized":"2024-09-22T03:35:17.564883+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564957+0000","last_change":"2024-09-22T03:35:17.569097+0000","last_active":"2024-09-22T03:35:17.564957+0000","last_peered":"2024-09-22T03:35:17.564957+0000","last_clean":"2024-09-22T03:35:17.564957+0000","last_became_active":"2024-09-22T03:35:15.604935+0000","last_became_peered":"2024-09-22T03:35:15.604935+0000","last_unstale":"2024-09-22T03:35:17.564957+0000","last_undegraded":"2024-09-22T03:35:17.564957+0000","last_fullsized":"2024-09-22T03:35:17.564957+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.565245+0000","last_change":"2024-09-22T03:35:17.568537+0000","last_active":"2024-09-22T03:35:17.565245+0000","last_peered":"2024-09-22T03:35:17.565245+0000","last_clean":"2024-09-22T03:35:17.565245+0000","last_became_active":"2024-09-22T03:35:15.604658+0000","last_became_peered":"2024-09-22T03:35:15.604658+0000","last_unstale":"2024-09-22T03:35:17.565245+0000","last_undegraded":"2024-09-22T03:35:17.565245+0000","last_fullsized":"2024-09-22T03:35:17.565245+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.565080+0000","last_change":"2024-09-22T03:35:17.569136+0000","last_active":"2024-09-22T03:35:17.565080+0000","last_peered":"2024-09-22T03:35:17.565080+0000","last_clean":"2024-09-22T03:35:17.565080+0000","last_became_active":"2024-09-22T03:35:15.606063+0000","last_became_peered":"2024-09-22T03:35:15.606063+0000","last_unstale":"2024-09-22T03:35:17.565080+0000","last_undegraded":"2024-09-22T03:35:17.565080+0000","last_fullsized":"2024-09-22T03:35:17.565080+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.579389+0000","last_change":"2024-09-22T03:35:17.579389+0000","last_active":"2024-09-22T03:35:17.579389+0000","last_peered":"2024-09-22T03:35:17.579389+0000","last_clean":"2024-09-22T03:35:17.579389+0000","last_became_active":"2024-09-22T03:35:15.603974+0000","last_became_peered":"2024-09-22T03:35:15.603974+0000","last_unstale":"2024-09-22T03:35:17.579389+0000","last_undegraded":"2024-09-22T03:35:17.579389+0000","last_fullsized":"2024-09-22T03:35:17.579389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.567689+0000","last_change":"2024-09-22T03:35:17.569319+0000","last_active":"2024-09-22T03:35:17.567689+0000","last_peered":"2024-09-22T03:35:17.567689+0000","last_clean":"2024-09-22T03:35:17.567689+0000","last_became_active":"2024-09-22T03:35:15.583736+0000","last_became_peered":"2024-09-22T03:35:15.583736+0000","last_unstale":"2024-09-22T03:35:17.567689+0000","last_undegraded":"2024-09-22T03:35:17.567689+0000","last_fullsized":"2024-09-22T03:35:17.567689+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564524+0000","last_change":"2024-09-22T03:35:17.567376+0000","last_active":"2024-09-22T03:35:17.564524+0000","last_peered":"2024-09-22T03:35:17.564524+0000","last_clean":"2024-09-22T03:35:17.564524+0000","last_became_active":"2024-09-22T03:35:15.579856+0000","last_became_peered":"2024-09-22T03:35:15.579856+0000","last_unstale":"2024-09-22T03:35:17.564524+0000","last_undegraded":"2024-09-22T03:35:17.564524+0000","last_fullsized":"2024-09-22T03:35:17.564524+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564977+0000","last_change":"2024-09-22T03:35:17.568473+0000","last_active":"2024-09-22T03:35:17.564977+0000","last_peered":"2024-09-22T03:35:17.564977+0000","last_clean":"2024-09-22T03:35:17.564977+0000","last_became_active":"2024-09-22T03:35:15.575073+0000","last_became_peered":"2024-09-22T03:35:15.575073+0000","last_unstale":"2024-09-22T03:35:17.564977+0000","last_undegraded":"2024-09-22T03:35:17.564977+0000","last_fullsized":"2024-09-22T03:35:17.564977+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564373+0000","last_change":"2024-09-22T03:35:14.604455+0000","last_active":"2024-09-22T03:35:17.564373+0000","last_peered":"2024-09-22T03:35:17.564373+0000","last_clean":"2024-09-22T03:35:17.564373+0000","last_became_active":"2024-09-22T03:35:14.590526+0000","last_became_peered":"2024-09-22T03:35:14.590526+0000","last_unstale":"2024-09-22T03:35:17.564373+0000","last_undegraded":"2024-09-22T03:35:17.564373+0000","last_fullsized":"2024-09-22T03:35:17.564373+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:13.533118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:13.533118+0000","last_clean_scrub_stamp":"2024-09-22T03:35:13.533118+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551218688,"internally_reserved":0,"allocated":1654784,"data_stored":1654784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":456,"kb_used_data":456,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297272,"statfs":{"total":53552873472,"available":53552406528,"internally_reserved":0,"allocated":466944,"data_stored":466944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1616,"kb_used_data":1616,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296112,"statfs":{"total":53552873472,"available":53551218688,"internally_reserved":0,"allocated":1654784,"data_stored":1654784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1360,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296368,"statfs":{"total":53552873472,"available":53551480832,"internally_reserved":0,"allocated":1392640,"data_stored":1392640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551218688,"internally_reserved":0,"allocated":1654784,"data_stored":1654784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551480832,"internally_reserved":0,"allocated":1392640,"data_stored":1392640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551218688,"internally_reserved":0,"allocated":1654784,"data_stored":1654784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552406528,"internally_reserved":0,"allocated":466944,"data_stored":466944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:35:19.386 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:35:19.490 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:19.485+0000 7fa8d4b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:19.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:19.490+0000 7fa8d4b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:19.665 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:35:19.666 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2024-09-22T03:35:19.677 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T03:35:19.478642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.835143"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564883+0000","last_change":"2024-09-22T03:35:17.568976+0000","last_active":"2024-09-22T03:35:17.564883+0000","last_peered":"2024-09-22T03:35:17.564883+0000","last_clean":"2024-09-22T03:35:17.564883+0000","last_became_active":"2024-09-22T03:35:15.611852+0000","last_became_peered":"2024-09-22T03:35:15.611852+0000","last_unstale":"2024-09-22T03:35:17.564883+0000","last_undegraded":"2024-09-22T03:35:17.564883+0000","last_fullsized":"2024-09-22T03:35:17.564883+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564957+0000","last_change":"2024-09-22T03:35:17.569097+0000","last_active":"2024-09-22T03:35:17.564957+0000","last_peered":"2024-09-22T03:35:17.564957+0000","last_clean":"2024-09-22T03:35:17.564957+0000","last_became_active":"2024-09-22T03:35:15.604935+0000","last_became_peered":"2024-09-22T03:35:15.604935+0000","last_unstale":"2024-09-22T03:35:17.564957+0000","last_undegraded":"2024-09-22T03:35:17.564957+0000","last_fullsized":"2024-09-22T03:35:17.564957+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.565245+0000","last_change":"2024-09-22T03:35:17.568537+0000","last_active":"2024-09-22T03:35:17.565245+0000","last_peered":"2024-09-22T03:35:17.565245+0000","last_clean":"2024-09-22T03:35:17.565245+0000","last_became_active":"2024-09-22T03:35:15.604658+0000","last_became_peered":"2024-09-22T03:35:15.604658+0000","last_unstale":"2024-09-22T03:35:17.565245+0000","last_undegraded":"2024-09-22T03:35:17.565245+0000","last_fullsized":"2024-09-22T03:35:17.565245+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.565080+0000","last_change":"2024-09-22T03:35:17.569136+0000","last_active":"2024-09-22T03:35:17.565080+0000","last_peered":"2024-09-22T03:35:17.565080+0000","last_clean":"2024-09-22T03:35:17.565080+0000","last_became_active":"2024-09-22T03:35:15.606063+0000","last_became_peered":"2024-09-22T03:35:15.606063+0000","last_unstale":"2024-09-22T03:35:17.565080+0000","last_undegraded":"2024-09-22T03:35:17.565080+0000","last_fullsized":"2024-09-22T03:35:17.565080+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.579389+0000","last_change":"2024-09-22T03:35:17.579389+0000","last_active":"2024-09-22T03:35:17.579389+0000","last_peered":"2024-09-22T03:35:17.579389+0000","last_clean":"2024-09-22T03:35:17.579389+0000","last_became_active":"2024-09-22T03:35:15.603974+0000","last_became_peered":"2024-09-22T03:35:15.603974+0000","last_unstale":"2024-09-22T03:35:17.579389+0000","last_undegraded":"2024-09-22T03:35:17.579389+0000","last_fullsized":"2024-09-22T03:35:17.579389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.567689+0000","last_change":"2024-09-22T03:35:17.569319+0000","last_active":"2024-09-22T03:35:17.567689+0000","last_peered":"2024-09-22T03:35:17.567689+0000","last_clean":"2024-09-22T03:35:17.567689+0000","last_became_active":"2024-09-22T03:35:15.583736+0000","last_became_peered":"2024-09-22T03:35:15.583736+0000","last_unstale":"2024-09-22T03:35:17.567689+0000","last_undegraded":"2024-09-22T03:35:17.567689+0000","last_fullsized":"2024-09-22T03:35:17.567689+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564524+0000","last_change":"2024-09-22T03:35:17.567376+0000","last_active":"2024-09-22T03:35:17.564524+0000","last_peered":"2024-09-22T03:35:17.564524+0000","last_clean":"2024-09-22T03:35:17.564524+0000","last_became_active":"2024-09-22T03:35:15.579856+0000","last_became_peered":"2024-09-22T03:35:15.579856+0000","last_unstale":"2024-09-22T03:35:17.564524+0000","last_undegraded":"2024-09-22T03:35:17.564524+0000","last_fullsized":"2024-09-22T03:35:17.564524+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564977+0000","last_change":"2024-09-22T03:35:17.568473+0000","last_active":"2024-09-22T03:35:17.564977+0000","last_peered":"2024-09-22T03:35:17.564977+0000","last_clean":"2024-09-22T03:35:17.564977+0000","last_became_active":"2024-09-22T03:35:15.575073+0000","last_became_peered":"2024-09-22T03:35:15.575073+0000","last_unstale":"2024-09-22T03:35:17.564977+0000","last_undegraded":"2024-09-22T03:35:17.564977+0000","last_fullsized":"2024-09-22T03:35:17.564977+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:14.537371+0000","last_clean_scrub_stamp":"2024-09-22T03:35:14.537371+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:35:17.564373+0000","last_change":"2024-09-22T03:35:14.604455+0000","last_active":"2024-09-22T03:35:17.564373+0000","last_peered":"2024-09-22T03:35:17.564373+0000","last_clean":"2024-09-22T03:35:17.564373+0000","last_became_active":"2024-09-22T03:35:14.590526+0000","last_became_peered":"2024-09-22T03:35:14.590526+0000","last_unstale":"2024-09-22T03:35:17.564373+0000","last_undegraded":"2024-09-22T03:35:17.564373+0000","last_fullsized":"2024-09-22T03:35:17.564373+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:35:13.533118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:35:13.533118+0000","last_clean_scrub_stamp":"2024-09-22T03:35:13.533118+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551218688,"internally_reserved":0,"allocated":1654784,"data_stored":1654784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":456,"kb_used_data":456,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297272,"statfs":{"total":53552873472,"available":53552406528,"internally_reserved":0,"allocated":466944,"data_stored":466944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1616,"kb_used_data":1616,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296112,"statfs":{"total":53552873472,"available":53551218688,"internally_reserved":0,"allocated":1654784,"data_stored":1654784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1360,"kb_used_data":1360,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296368,"statfs":{"total":53552873472,"available":53551480832,"internally_reserved":0,"allocated":1392640,"data_stored":1392640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551218688,"internally_reserved":0,"allocated":1654784,"data_stored":1654784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551480832,"internally_reserved":0,"allocated":1392640,"data_stored":1392640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551218688,"internally_reserved":0,"allocated":1654784,"data_stored":1654784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552406528,"internally_reserved":0,"allocated":466944,"data_stored":466944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:35:19.678 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:35:19.678 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T03:35:19.678 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T03:35:19.678 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T03:35:19.780 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:19.776+0000 7f4a2b397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:19.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:19.780+0000 7f4a2b397640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:19.992 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-09-22T03:35:19.993 INFO:teuthology.orchestra.run.smithi122.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T03:35:20.005 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T03:35:20.005 INFO:teuthology.run_tasks:Running task radosbench... 2024-09-22T03:35:20.015 INFO:tasks.radosbench:Beginning radosbench... 2024-09-22T03:35:20.015 INFO:tasks.radosbench:omap writes 2024-09-22T03:35:20.015 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-22T03:35:20.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-22T03:35:20.121 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:20.114+0000 7f9cd0197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:20.123 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:20.122+0000 7f9cd0197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:20.653 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'unique_pool_0' created 2024-09-22T03:35:20.672 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-22T03:35:20.800 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:20.790+0000 7f2e58317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:20.802 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:35:20.801+0000 7f2e58317640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:21.658 INFO:teuthology.orchestra.run.smithi122.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-22T03:35:22.681 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 4096 --object-size 4096 -p unique_pool_0 bench 300 write --write-omap' 2024-09-22T03:35:22.684 DEBUG:teuthology.run_tasks:Unwinding manager radosbench 2024-09-22T03:35:22.694 INFO:tasks.radosbench:joining radosbench (timing out after 9300s) 2024-09-22T03:35:22.694 INFO:teuthology.orchestra.run:waiting for 9300 2024-09-22T03:35:22.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:hints = 1 2024-09-22T03:35:22.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 300 seconds or 0 objects 2024-09-22T03:35:22.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Object prefix: benchmark_data_smithi122_29430 2024-09-22T03:35:22.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:35:22.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-09-22T03:35:23.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 1 16 196 180 0.703083 0.703125 0.0671182 0.0675546 2024-09-22T03:35:24.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 2 16 402 386 0.753852 0.804688 0.0778636 0.0780673 2024-09-22T03:35:25.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 3 16 637 621 0.808532 0.917969 0.0270905 0.0744729 2024-09-22T03:35:26.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 4 16 868 852 0.831966 0.902344 0.0153474 0.0731412 2024-09-22T03:35:27.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 5 16 1091 1075 0.839777 0.871094 0.0166331 0.0718423 2024-09-22T03:35:28.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 6 16 1298 1282 0.834565 0.808594 0.0141332 0.0737871 2024-09-22T03:35:29.739 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 7 16 1543 1527 0.852049 0.957031 0.0400688 0.0724676 2024-09-22T03:35:30.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 8 16 1778 1762 0.860277 0.917969 0.038985 0.0721867 2024-09-22T03:35:31.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 9 16 2029 2013 0.873622 0.980469 0.0715629 0.070868 2024-09-22T03:35:32.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 10 16 2273 2257 0.881562 0.953125 0.0100939 0.0702098 2024-09-22T03:35:33.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 11 16 2539 2523 0.895872 1.03906 0.0322804 0.0692886 2024-09-22T03:35:34.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 12 16 2801 2785 0.906495 1.02344 0.131535 0.0685799 2024-09-22T03:35:35.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 13 16 3062 3046 0.915182 1.01953 0.125427 0.0679503 2024-09-22T03:35:36.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 14 15 3314 3299 0.920398 0.988281 0.0382869 0.0676259 2024-09-22T03:35:37.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 15 16 3571 3555 0.9257 1 0.0259175 0.0671153 2024-09-22T03:35:38.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 16 16 3795 3779 0.922526 0.875 0.023984 0.067519 2024-09-22T03:35:39.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 17 16 4023 4007 0.920644 0.890625 0.0288411 0.0677033 2024-09-22T03:35:40.740 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 18 16 4228 4212 0.913981 0.800781 0.0790625 0.0681355 2024-09-22T03:35:41.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 19 16 4420 4404 0.905347 0.75 0.0405166 0.0687075 2024-09-22T03:35:42.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:35:42.740738+0000 min lat: 0.00649382 max lat: 0.754414 avg lat: 0.06935 2024-09-22T03:35:42.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:35:42.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 20 16 4602 4586 0.895625 0.710938 0.0135885 0.06935 2024-09-22T03:35:43.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 21 16 4802 4786 0.890174 0.78125 0.040011 0.0699349 2024-09-22T03:35:44.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 22 16 5003 4987 0.885397 0.785156 0.0288589 0.0701723 2024-09-22T03:35:45.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 23 16 5195 5179 0.879507 0.75 0.0545971 0.0707705 2024-09-22T03:35:46.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 24 16 5401 5385 0.876387 0.804688 0.106138 0.0711693 2024-09-22T03:35:47.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 25 16 5602 5586 0.872736 0.785156 0.0154549 0.0713549 2024-09-22T03:35:48.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 26 16 5788 5772 0.867112 0.726562 0.039279 0.0718594 2024-09-22T03:35:49.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 27 16 5983 5967 0.863206 0.761719 0.0976924 0.0722198 2024-09-22T03:35:50.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 28 16 6192 6176 0.861532 0.816406 0.0480987 0.0724154 2024-09-22T03:35:51.741 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 29 16 6384 6368 0.857683 0.75 0.0241418 0.0726336 2024-09-22T03:35:52.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 30 16 6592 6576 0.856174 0.8125 0.0508753 0.0727767 2024-09-22T03:35:53.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 31 16 6778 6762 0.851991 0.726562 0.04525 0.0731592 2024-09-22T03:35:54.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 32 16 6950 6934 0.84636 0.671875 0.0247812 0.073581 2024-09-22T03:35:55.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 33 16 7154 7138 0.844859 0.796875 0.0783387 0.073885 2024-09-22T03:35:56.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 34 16 7353 7337 0.842871 0.777344 0.0316383 0.073987 2024-09-22T03:35:57.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 35 16 7557 7541 0.841555 0.796875 0.0383762 0.0740099 2024-09-22T03:35:58.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 36 16 7760 7744 0.840204 0.792969 0.148083 0.0742872 2024-09-22T03:35:59.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 37 16 7932 7916 0.835653 0.671875 0.0378778 0.0745597 2024-09-22T03:36:00.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 38 16 8113 8097 0.832267 0.707031 0.0389013 0.0749474 2024-09-22T03:36:01.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 39 16 8298 8282 0.829454 0.722656 0.0387174 0.0751856 2024-09-22T03:36:02.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:36:02.742492+0000 min lat: 0.00649382 max lat: 0.754414 avg lat: 0.0754561 2024-09-22T03:36:02.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:36:02.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 40 16 8471 8455 0.825611 0.675781 0.06066 0.0754561 2024-09-22T03:36:03.742 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 41 16 8654 8638 0.822908 0.714844 0.0306417 0.0758226 2024-09-22T03:36:04.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 42 16 8836 8820 0.820241 0.710938 0.0445118 0.0760563 2024-09-22T03:36:05.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 43 16 9000 8984 0.816062 0.640625 0.10729 0.0764355 2024-09-22T03:36:06.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 44 16 9186 9170 0.814027 0.726562 0.0321414 0.0766853 2024-09-22T03:36:07.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 45 16 9372 9356 0.812082 0.726562 0.0490058 0.0767663 2024-09-22T03:36:08.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 46 16 9573 9557 0.811495 0.785156 0.104268 0.0768413 2024-09-22T03:36:09.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 47 16 9745 9729 0.808523 0.671875 0.0355568 0.0771611 2024-09-22T03:36:10.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 48 16 9918 9902 0.805757 0.675781 0.0698863 0.0774144 2024-09-22T03:36:11.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 49 16 10093 10077 0.803262 0.683594 0.0626957 0.0776557 2024-09-22T03:36:12.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 50 16 10282 10266 0.801961 0.738281 0.424129 0.0777505 2024-09-22T03:36:13.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 51 16 10465 10449 0.800252 0.714844 0.0432918 0.0779699 2024-09-22T03:36:14.743 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 52 16 10638 10622 0.797857 0.675781 0.328678 0.0781632 2024-09-22T03:36:15.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 53 16 10823 10807 0.796437 0.722656 0.0434201 0.0782923 2024-09-22T03:36:16.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 54 16 10998 10982 0.794346 0.683594 0.0411572 0.0785091 2024-09-22T03:36:17.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 55 16 11181 11165 0.792899 0.714844 0.0413391 0.0786939 2024-09-22T03:36:18.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 56 15 11355 11340 0.790946 0.683594 0.0483943 0.0789039 2024-09-22T03:36:19.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 57 16 11514 11498 0.787897 0.617188 0.0369033 0.0790976 2024-09-22T03:36:20.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 58 16 11698 11682 0.786704 0.71875 0.0579731 0.0793832 2024-09-22T03:36:21.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 59 16 11877 11861 0.78522 0.699219 0.0180228 0.0795245 2024-09-22T03:36:22.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:36:22.744234+0000 min lat: 0.00649382 max lat: 0.754414 avg lat: 0.0796088 2024-09-22T03:36:22.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:36:22.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 60 16 12050 12034 0.783395 0.675781 0.0322498 0.0796088 2024-09-22T03:36:23.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 61 16 12220 12204 0.781438 0.664062 0.234773 0.0798312 2024-09-22T03:36:24.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 62 16 12398 12382 0.780047 0.695312 0.350817 0.0799782 2024-09-22T03:36:25.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 63 16 12559 12543 0.777647 0.628906 0.0409722 0.0801938 2024-09-22T03:36:26.744 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 64 16 12707 12691 0.774529 0.578125 0.0270637 0.0804905 2024-09-22T03:36:27.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 65 16 12879 12863 0.772949 0.671875 0.0175296 0.0807461 2024-09-22T03:36:28.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 66 16 13040 13024 0.770765 0.628906 0.0658872 0.0809573 2024-09-22T03:36:29.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 67 16 13197 13181 0.768414 0.613281 0.0196722 0.0811881 2024-09-22T03:36:30.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 68 16 13355 13339 0.766189 0.617188 0.0484487 0.0814887 2024-09-22T03:36:31.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 69 16 13507 13491 0.763689 0.59375 0.165827 0.0817251 2024-09-22T03:36:32.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 70 16 13668 13652 0.761763 0.628906 0.0188255 0.0818736 2024-09-22T03:36:33.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 71 16 13828 13812 0.759836 0.625 0.197376 0.0821187 2024-09-22T03:36:34.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 72 16 13990 13974 0.758071 0.632812 0.058768 0.0823091 2024-09-22T03:36:35.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 73 16 14161 14145 0.756836 0.667969 0.0950328 0.0824816 2024-09-22T03:36:36.745 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 74 16 14335 14319 0.755792 0.679688 0.0220344 0.082611 2024-09-22T03:36:37.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 75 16 14522 14506 0.755454 0.730469 0.0475608 0.0826242 2024-09-22T03:36:38.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 76 16 14685 14669 0.753891 0.636719 0.0311737 0.0827876 2024-09-22T03:36:39.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 77 16 14843 14827 0.752114 0.617188 0.176139 0.0830023 2024-09-22T03:36:40.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 78 16 15010 14994 0.750834 0.652344 0.0238167 0.0830656 2024-09-22T03:36:41.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 79 16 15163 15147 0.748895 0.597656 0.024372 0.0833002 2024-09-22T03:36:42.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:36:42.746156+0000 min lat: 0.00649382 max lat: 0.754414 avg lat: 0.083562 2024-09-22T03:36:42.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:36:42.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 80 16 15311 15295 0.746759 0.578125 0.512888 0.083562 2024-09-22T03:36:43.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 81 16 15474 15458 0.7454 0.636719 0.189623 0.0837332 2024-09-22T03:36:44.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 82 16 15635 15619 0.743979 0.628906 0.225794 0.0838592 2024-09-22T03:36:45.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 83 16 15795 15779 0.742544 0.625 0.342036 0.0840599 2024-09-22T03:36:46.746 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 84 16 15952 15936 0.741005 0.613281 0.0520944 0.0842333 2024-09-22T03:36:47.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 85 16 16100 16084 0.739088 0.578125 0.0455279 0.0844509 2024-09-22T03:36:48.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 86 16 16263 16247 0.737897 0.636719 0.251619 0.0846195 2024-09-22T03:36:49.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 87 16 16426 16410 0.736733 0.636719 0.274816 0.0847227 2024-09-22T03:36:50.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 88 16 16578 16562 0.735108 0.59375 0.242608 0.0849519 2024-09-22T03:36:51.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 89 16 16733 16717 0.733651 0.605469 0.484391 0.0850531 2024-09-22T03:36:52.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 90 16 16892 16876 0.732399 0.621094 0.0491477 0.0852411 2024-09-22T03:36:53.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 91 16 17044 17028 0.730875 0.59375 0.0288936 0.0853859 2024-09-22T03:36:54.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 92 16 17202 17186 0.729639 0.617188 0.0382734 0.0855092 2024-09-22T03:36:55.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 93 16 17368 17352 0.728765 0.648438 0.0546138 0.0856898 2024-09-22T03:36:56.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 94 16 17514 17498 0.727079 0.570312 0.430573 0.0858284 2024-09-22T03:36:57.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 95 16 17671 17655 0.725881 0.613281 0.169727 0.0860219 2024-09-22T03:36:58.747 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 96 16 17807 17791 0.723853 0.53125 0.0177797 0.0862032 2024-09-22T03:36:59.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 97 16 17954 17938 0.72231 0.574219 0.0331007 0.0864179 2024-09-22T03:37:00.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 98 16 18105 18089 0.720957 0.589844 0.0544151 0.0866351 2024-09-22T03:37:01.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 99 16 18250 18234 0.719396 0.566406 0.0304148 0.0867621 2024-09-22T03:37:02.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:37:02.747948+0000 min lat: 0.00649382 max lat: 0.754414 avg lat: 0.0869054 2024-09-22T03:37:02.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:37:02.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 100 16 18412 18396 0.718529 0.632812 0.100253 0.0869054 2024-09-22T03:37:03.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 101 16 18563 18547 0.717255 0.589844 0.0694344 0.0870415 2024-09-22T03:37:04.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 102 16 18729 18713 0.716579 0.648438 0.0933221 0.0871383 2024-09-22T03:37:05.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 103 16 18884 18868 0.7155 0.605469 0.0395489 0.0872737 2024-09-22T03:37:06.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 104 16 19031 19015 0.714141 0.574219 0.0729477 0.0874389 2024-09-22T03:37:07.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 105 16 19177 19161 0.712771 0.570312 0.066332 0.0875574 2024-09-22T03:37:08.748 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 106 16 19324 19308 0.711463 0.574219 0.0406172 0.0877535 2024-09-22T03:37:09.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 107 16 19470 19454 0.710143 0.570312 0.17303 0.0879046 2024-09-22T03:37:10.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 108 16 19616 19600 0.708848 0.570312 0.0174511 0.0880386 2024-09-22T03:37:11.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 109 16 19750 19734 0.707147 0.523438 0.0849787 0.0882858 2024-09-22T03:37:12.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 110 16 19917 19901 0.706648 0.652344 0.301265 0.0883554 2024-09-22T03:37:13.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 111 16 20062 20046 0.705384 0.566406 0.521592 0.0884625 2024-09-22T03:37:14.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 112 16 20195 20179 0.703724 0.519531 0.726717 0.0886674 2024-09-22T03:37:15.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 113 16 20346 20330 0.702716 0.589844 0.0224129 0.0888271 2024-09-22T03:37:16.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 114 16 20504 20488 0.701965 0.617188 0.346762 0.0889627 2024-09-22T03:37:17.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 115 16 20651 20635 0.700854 0.574219 0.0282758 0.0890468 2024-09-22T03:37:18.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 116 16 20812 20796 0.700233 0.628906 0.0954024 0.089191 2024-09-22T03:37:19.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 117 16 20959 20943 0.699156 0.574219 0.0345931 0.0892944 2024-09-22T03:37:20.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 118 16 21097 21081 0.697799 0.539062 0.0725145 0.0894915 2024-09-22T03:37:21.749 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 119 16 21246 21230 0.696826 0.582031 0.0523195 0.089617 2024-09-22T03:37:22.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:37:22.749756+0000 min lat: 0.00649382 max lat: 0.758387 avg lat: 0.0897135 2024-09-22T03:37:22.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:37:22.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 120 16 21377 21361 0.695283 0.511719 0.0387622 0.0897135 2024-09-22T03:37:23.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 121 16 21538 21522 0.694733 0.628906 0.0666021 0.0898853 2024-09-22T03:37:24.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 122 16 21685 21669 0.693745 0.574219 0.0731558 0.0900277 2024-09-22T03:37:25.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 123 16 21830 21814 0.69271 0.566406 0.0609387 0.0901652 2024-09-22T03:37:26.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 124 16 21969 21953 0.691502 0.542969 0.0299366 0.0903054 2024-09-22T03:37:27.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 125 16 22120 22104 0.690688 0.589844 0.0899297 0.0904297 2024-09-22T03:37:28.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 126 16 22288 22272 0.690414 0.65625 0.115792 0.0904785 2024-09-22T03:37:29.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 127 16 22429 22413 0.689314 0.550781 0.0691012 0.0905857 2024-09-22T03:37:30.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 128 16 22563 22547 0.688018 0.523438 0.0280133 0.0907441 2024-09-22T03:37:31.750 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 129 16 22705 22689 0.686984 0.554688 0.0743682 0.0908974 2024-09-22T03:37:32.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 130 16 22862 22846 0.686417 0.613281 0.274983 0.0909939 2024-09-22T03:37:33.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 131 16 23002 22986 0.685351 0.546875 0.0345674 0.0911262 2024-09-22T03:37:34.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 132 16 23141 23125 0.684272 0.542969 0.195211 0.091276 2024-09-22T03:37:35.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 133 16 23290 23274 0.683503 0.582031 0.110128 0.0913928 2024-09-22T03:37:36.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 134 16 23448 23432 0.683008 0.617188 0.0758988 0.0914359 2024-09-22T03:37:37.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 135 16 23592 23576 0.682115 0.5625 0.0669293 0.0915522 2024-09-22T03:37:38.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 136 16 23734 23718 0.681177 0.554688 0.0619069 0.0916656 2024-09-22T03:37:39.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 137 16 23876 23860 0.680253 0.554688 0.0683594 0.091801 2024-09-22T03:37:40.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 138 16 24028 24012 0.679626 0.59375 0.133704 0.091923 2024-09-22T03:37:41.751 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 139 16 24166 24150 0.678615 0.539062 0.106764 0.0920542 2024-09-22T03:37:42.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:37:42.751645+0000 min lat: 0.00649382 max lat: 0.826036 avg lat: 0.0921443 2024-09-22T03:37:42.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:37:42.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 140 16 24306 24290 0.677673 0.546875 0.0890351 0.0921443 2024-09-22T03:37:43.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 141 16 24443 24427 0.676662 0.535156 0.337548 0.0923058 2024-09-22T03:37:44.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 142 16 24581 24565 0.675692 0.539062 0.0613438 0.0923976 2024-09-22T03:37:45.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 143 16 24718 24702 0.674709 0.535156 0.0352256 0.0925448 2024-09-22T03:37:46.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 144 16 24848 24832 0.67355 0.507812 0.0451369 0.0926906 2024-09-22T03:37:47.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 145 16 24985 24969 0.672595 0.535156 0.0286138 0.0928004 2024-09-22T03:37:48.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 146 16 25129 25113 0.671841 0.5625 0.187735 0.0929855 2024-09-22T03:37:49.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 147 16 25260 25244 0.670751 0.511719 0.146671 0.0931092 2024-09-22T03:37:50.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 148 16 25402 25386 0.669967 0.554688 0.520704 0.0932144 2024-09-22T03:37:51.752 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 149 16 25540 25524 0.669088 0.539062 0.101169 0.0933318 2024-09-22T03:37:52.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 150 16 25675 25659 0.668142 0.527344 0.0706573 0.0934489 2024-09-22T03:37:53.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 151 16 25819 25803 0.667442 0.5625 0.0504613 0.0935458 2024-09-22T03:37:54.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 152 16 25960 25944 0.666675 0.550781 0.0439232 0.0936544 2024-09-22T03:37:55.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 153 16 26088 26072 0.665585 0.5 0.0963244 0.0937994 2024-09-22T03:37:56.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 154 16 26234 26218 0.664966 0.570312 0.083007 0.0939369 2024-09-22T03:37:57.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 155 16 26363 26347 0.663926 0.503906 0.0578615 0.0940667 2024-09-22T03:37:58.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 156 16 26505 26489 0.663226 0.554688 0.326111 0.0941647 2024-09-22T03:37:59.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 157 16 26639 26623 0.662335 0.523438 0.0488499 0.0942906 2024-09-22T03:38:00.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 158 16 26768 26752 0.661332 0.503906 0.143052 0.0944057 2024-09-22T03:38:01.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 159 16 26907 26891 0.660588 0.542969 0.089938 0.0945492 2024-09-22T03:38:02.753 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:38:02.753559+0000 min lat: 0.00649382 max lat: 0.826036 avg lat: 0.0946598 2024-09-22T03:38:02.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:38:02.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 160 16 27039 27023 0.659681 0.515625 0.0206606 0.0946598 2024-09-22T03:38:03.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 161 16 27178 27162 0.658956 0.542969 0.0366484 0.0947826 2024-09-22T03:38:04.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 162 16 27322 27306 0.65836 0.5625 0.0618793 0.094873 2024-09-22T03:38:05.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 163 16 27455 27439 0.657508 0.519531 0.0495974 0.0949517 2024-09-22T03:38:06.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 164 16 27593 27577 0.656785 0.539062 0.097257 0.0951194 2024-09-22T03:38:07.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 165 16 27729 27713 0.656024 0.53125 0.0455177 0.0951803 2024-09-22T03:38:08.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 166 16 27863 27847 0.655225 0.523438 0.162605 0.0953345 2024-09-22T03:38:09.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 167 16 28000 27984 0.654506 0.535156 0.022175 0.0954041 2024-09-22T03:38:10.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 168 16 28131 28115 0.653656 0.511719 0.0878431 0.0955569 2024-09-22T03:38:11.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 169 16 28255 28239 0.652654 0.484375 0.0459764 0.095665 2024-09-22T03:38:12.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 170 16 28383 28367 0.651756 0.5 0.0461216 0.0958259 2024-09-22T03:38:13.754 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 171 16 28515 28499 0.650959 0.515625 0.0316457 0.0959537 2024-09-22T03:38:14.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 172 16 28649 28633 0.650218 0.523438 0.0759552 0.0960571 2024-09-22T03:38:15.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 173 16 28783 28767 0.649484 0.523438 0.0506547 0.0961556 2024-09-22T03:38:16.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 174 16 28917 28901 0.64876 0.523438 0.0833755 0.0962322 2024-09-22T03:38:17.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 175 16 29049 29033 0.647999 0.515625 0.0291079 0.0963679 2024-09-22T03:38:18.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 176 16 29178 29162 0.64718 0.503906 0.128576 0.0965231 2024-09-22T03:38:19.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 177 16 29307 29291 0.64637 0.503906 0.206243 0.0966184 2024-09-22T03:38:20.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 178 16 29441 29425 0.645679 0.523438 0.0800727 0.0967622 2024-09-22T03:38:21.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 179 16 29557 29541 0.644603 0.453125 0.0399375 0.0968424 2024-09-22T03:38:22.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:38:22.755499+0000 min lat: 0.00649382 max lat: 0.826036 avg lat: 0.0969868 2024-09-22T03:38:22.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:38:22.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 180 16 29699 29683 0.644103 0.554688 0.0549666 0.0969868 2024-09-22T03:38:23.755 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 181 16 29825 29809 0.643264 0.492188 0.0335343 0.0970572 2024-09-22T03:38:24.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 182 16 29954 29938 0.642498 0.503906 0.0883579 0.0971893 2024-09-22T03:38:25.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 183 16 30081 30065 0.641697 0.496094 0.0424591 0.0973145 2024-09-22T03:38:26.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 184 16 30218 30202 0.641118 0.535156 0.0502036 0.097426 2024-09-22T03:38:27.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 185 16 30364 30348 0.640735 0.570312 0.191482 0.097491 2024-09-22T03:38:28.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 186 16 30484 30468 0.63981 0.46875 0.119832 0.0976112 2024-09-22T03:38:29.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 187 16 30611 30595 0.639041 0.496094 0.0366489 0.0977196 2024-09-22T03:38:30.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 188 16 30736 30720 0.638239 0.488281 0.0434569 0.0978076 2024-09-22T03:38:31.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 189 16 30864 30848 0.637508 0.5 0.121575 0.0979535 2024-09-22T03:38:32.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 190 16 30996 30980 0.636866 0.515625 0.0722203 0.0980706 2024-09-22T03:38:33.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 191 16 31135 31119 0.636374 0.542969 0.128481 0.098167 2024-09-22T03:38:34.756 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 192 16 31266 31250 0.635725 0.511719 0.119844 0.0982771 2024-09-22T03:38:35.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 193 16 31377 31361 0.634677 0.433594 0.0383647 0.0983792 2024-09-22T03:38:36.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 194 16 31508 31492 0.634043 0.511719 0.0469476 0.0984866 2024-09-22T03:38:37.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 195 16 31623 31607 0.633095 0.449219 0.535606 0.0986285 2024-09-22T03:38:38.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 196 16 31748 31732 0.632356 0.488281 0.0533211 0.0987846 2024-09-22T03:38:39.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 197 16 31900 31884 0.632159 0.59375 0.0478254 0.098835 2024-09-22T03:38:40.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 198 16 32024 32008 0.631413 0.484375 0.0690786 0.0989411 2024-09-22T03:38:41.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 199 16 32156 32140 0.630831 0.515625 0.0585701 0.0990308 2024-09-22T03:38:42.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:38:42.757397+0000 min lat: 0.00649382 max lat: 0.826036 avg lat: 0.0991066 2024-09-22T03:38:42.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:38:42.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 200 16 32290 32274 0.630294 0.523438 0.0740542 0.0991066 2024-09-22T03:38:43.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 201 16 32411 32395 0.629509 0.472656 0.0664657 0.0991988 2024-09-22T03:38:44.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 202 16 32534 32518 0.628771 0.480469 0.0466679 0.0992772 2024-09-22T03:38:45.757 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 203 16 32656 32640 0.628021 0.476562 0.0244984 0.0994329 2024-09-22T03:38:46.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 204 16 32805 32789 0.627795 0.582031 0.160039 0.0995107 2024-09-22T03:38:47.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 205 16 32943 32927 0.627362 0.539062 0.0447813 0.0995727 2024-09-22T03:38:48.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 206 16 33072 33056 0.626763 0.503906 0.127157 0.0996633 2024-09-22T03:38:49.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 207 16 33216 33200 0.626452 0.5625 0.0428852 0.0997255 2024-09-22T03:38:50.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 208 16 33347 33331 0.6259 0.511719 0.148456 0.0998081 2024-09-22T03:38:51.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 209 16 33476 33460 0.625316 0.503906 0.037797 0.0998901 2024-09-22T03:38:52.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 210 16 33594 33578 0.624533 0.460938 0.0330728 0.0999856 2024-09-22T03:38:53.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 211 16 33727 33711 0.624035 0.519531 0.304024 0.100114 2024-09-22T03:38:54.758 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 212 16 33849 33833 0.62334 0.476562 0.496914 0.1002 2024-09-22T03:38:55.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 213 15 33967 33952 0.622595 0.464844 0.0451796 0.100301 2024-09-22T03:38:56.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 214 16 34097 34081 0.62204 0.503906 0.454143 0.100398 2024-09-22T03:38:57.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 215 16 34201 34185 0.621037 0.40625 0.0544126 0.10054 2024-09-22T03:38:58.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 216 16 34320 34304 0.620313 0.464844 0.107656 0.100689 2024-09-22T03:38:59.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 217 16 34439 34423 0.619597 0.464844 0.020369 0.1008 2024-09-22T03:39:00.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 218 16 34562 34546 0.618958 0.480469 0.0666845 0.100929 2024-09-22T03:39:01.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 219 16 34695 34679 0.618504 0.519531 0.053251 0.100974 2024-09-22T03:39:02.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:39:02.759262+0000 min lat: 0.00649382 max lat: 0.826036 avg lat: 0.101119 2024-09-22T03:39:02.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:39:02.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 220 16 34812 34796 0.61777 0.457031 0.0584624 0.101119 2024-09-22T03:39:03.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 221 16 34940 34924 0.617237 0.5 0.0531759 0.101185 2024-09-22T03:39:04.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 222 16 35062 35046 0.616603 0.476562 0.0365707 0.101295 2024-09-22T03:39:05.759 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 223 16 35189 35173 0.616062 0.496094 0.0428868 0.10139 2024-09-22T03:39:06.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 224 16 35308 35292 0.615387 0.464844 0.1459 0.101502 2024-09-22T03:39:07.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 225 16 35433 35417 0.614822 0.488281 0.177519 0.101602 2024-09-22T03:39:08.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 226 16 35553 35537 0.614175 0.46875 0.202334 0.101705 2024-09-22T03:39:09.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 227 16 35689 35673 0.61381 0.53125 0.0358465 0.101755 2024-09-22T03:39:10.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 228 16 35799 35783 0.613002 0.429688 0.0209801 0.101859 2024-09-22T03:39:11.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 229 16 35931 35915 0.612576 0.515625 0.0837155 0.101983 2024-09-22T03:39:12.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 230 16 36056 36040 0.612036 0.488281 0.0383236 0.102086 2024-09-22T03:39:13.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 231 16 36182 36166 0.611517 0.492188 0.0980827 0.102152 2024-09-22T03:39:14.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 232 16 36315 36299 0.61112 0.519531 0.0534985 0.102227 2024-09-22T03:39:15.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 233 16 36421 36405 0.610274 0.414062 0.0766643 0.10234 2024-09-22T03:39:16.760 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 234 16 36539 36523 0.609636 0.460938 0.154121 0.102447 2024-09-22T03:39:17.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 235 16 36645 36629 0.608803 0.414062 0.0544805 0.102567 2024-09-22T03:39:18.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 236 16 36768 36752 0.608259 0.480469 0.0172372 0.102694 2024-09-22T03:39:19.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 237 16 36888 36872 0.607671 0.46875 0.0653523 0.102798 2024-09-22T03:39:20.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 238 16 37012 36996 0.607152 0.484375 0.284702 0.102888 2024-09-22T03:39:21.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 239 16 37133 37117 0.606589 0.472656 0.0772097 0.102985 2024-09-22T03:39:22.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:39:22.761233+0000 min lat: 0.00649382 max lat: 0.864719 avg lat: 0.103078 2024-09-22T03:39:22.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:39:22.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 240 16 37258 37242 0.606096 0.488281 0.0717097 0.103078 2024-09-22T03:39:23.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 241 16 37384 37368 0.605623 0.492188 0.0320595 0.103139 2024-09-22T03:39:24.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 242 16 37509 37493 0.605138 0.488281 0.156115 0.103227 2024-09-22T03:39:25.761 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 243 16 37629 37613 0.604577 0.46875 0.184702 0.103315 2024-09-22T03:39:26.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 244 16 37746 37730 0.603972 0.457031 0.135197 0.103425 2024-09-22T03:39:27.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 245 16 37865 37849 0.603404 0.464844 0.0589591 0.103543 2024-09-22T03:39:28.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 246 16 37991 37975 0.602951 0.492188 0.132031 0.103597 2024-09-22T03:39:29.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 247 16 38119 38103 0.602534 0.5 0.0306901 0.103674 2024-09-22T03:39:30.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 248 16 38232 38216 0.601885 0.441406 0.0675355 0.103783 2024-09-22T03:39:31.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 249 16 38351 38335 0.601334 0.464844 0.0312717 0.103874 2024-09-22T03:39:32.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 250 16 38470 38454 0.600788 0.464844 0.149029 0.103969 2024-09-22T03:39:33.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 251 16 38606 38590 0.600511 0.53125 0.12153 0.10404 2024-09-22T03:39:34.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 252 16 38718 38702 0.599864 0.4375 0.0871214 0.104114 2024-09-22T03:39:35.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 253 16 38834 38818 0.599283 0.453125 0.0460197 0.104216 2024-09-22T03:39:36.762 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 254 16 38965 38949 0.598939 0.511719 0.184264 0.10432 2024-09-22T03:39:37.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 255 16 39094 39078 0.598566 0.503906 0.0796037 0.104364 2024-09-22T03:39:38.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 256 16 39215 39199 0.598074 0.472656 0.0808633 0.104457 2024-09-22T03:39:39.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 257 16 39331 39315 0.597509 0.453125 0.0877459 0.10456 2024-09-22T03:39:40.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 258 16 39450 39434 0.596995 0.464844 0.238094 0.104636 2024-09-22T03:39:41.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 259 16 39572 39556 0.59653 0.476562 0.0557057 0.104728 2024-09-22T03:39:42.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:39:42.763187+0000 min lat: 0.00649382 max lat: 0.864719 avg lat: 0.104769 2024-09-22T03:39:42.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:39:42.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 260 16 39713 39697 0.596354 0.550781 0.162736 0.104769 2024-09-22T03:39:43.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 261 16 39840 39824 0.595969 0.496094 0.0333444 0.104827 2024-09-22T03:39:44.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 262 16 39974 39958 0.595692 0.523438 0.132337 0.104881 2024-09-22T03:39:45.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 263 16 40093 40077 0.595195 0.464844 0.176762 0.104963 2024-09-22T03:39:46.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 264 16 40197 40181 0.594479 0.40625 0.459171 0.105088 2024-09-22T03:39:47.763 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 265 16 40320 40304 0.594048 0.480469 0.0591941 0.105174 2024-09-22T03:39:48.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 266 16 40440 40424 0.593577 0.46875 0.0786492 0.105226 2024-09-22T03:39:49.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 267 16 40553 40537 0.593007 0.441406 0.170517 0.105331 2024-09-22T03:39:50.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 268 16 40679 40663 0.592631 0.492188 0.0421867 0.105404 2024-09-22T03:39:51.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 269 16 40794 40778 0.592097 0.449219 0.0412709 0.105489 2024-09-22T03:39:52.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 270 16 40910 40894 0.591583 0.453125 0.278906 0.105599 2024-09-22T03:39:53.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 271 16 41018 41002 0.590956 0.421875 0.238192 0.105711 2024-09-22T03:39:54.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 272 16 41135 41119 0.590464 0.457031 0.0766683 0.105815 2024-09-22T03:39:55.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 273 16 41260 41244 0.590089 0.488281 0.0678653 0.105858 2024-09-22T03:39:56.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 274 16 41385 41369 0.589718 0.488281 0.0907653 0.105948 2024-09-22T03:39:57.764 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 275 16 41505 41489 0.589277 0.46875 0.149559 0.10601 2024-09-22T03:39:58.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 276 16 41621 41605 0.588784 0.453125 0.129882 0.106107 2024-09-22T03:39:59.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 277 16 41741 41725 0.58835 0.46875 0.0291869 0.106174 2024-09-22T03:40:00.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 278 16 41858 41842 0.587878 0.457031 0.16707 0.106259 2024-09-22T03:40:01.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 279 16 41977 41961 0.587437 0.464844 0.0457085 0.106346 2024-09-22T03:40:02.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:40:02.765104+0000 min lat: 0.00649382 max lat: 0.864719 avg lat: 0.106429 2024-09-22T03:40:02.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:40:02.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 280 16 42092 42076 0.586943 0.449219 0.130437 0.106429 2024-09-22T03:40:03.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 281 16 42207 42191 0.586453 0.449219 0.0196853 0.106494 2024-09-22T03:40:04.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 282 16 42321 42305 0.585952 0.445312 0.519083 0.106594 2024-09-22T03:40:05.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 283 16 42432 42416 0.585414 0.433594 0.0365863 0.106723 2024-09-22T03:40:06.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 284 16 42550 42534 0.584975 0.460938 0.0741802 0.10679 2024-09-22T03:40:07.765 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 285 16 42669 42653 0.584553 0.464844 0.0787675 0.106861 2024-09-22T03:40:08.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 286 16 42793 42777 0.584203 0.484375 0.131321 0.106918 2024-09-22T03:40:09.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 287 16 42899 42883 0.58361 0.414062 0.128263 0.107013 2024-09-22T03:40:10.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 288 16 43014 42998 0.583143 0.449219 0.0543664 0.107132 2024-09-22T03:40:11.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 289 16 43125 43109 0.582626 0.433594 0.0640617 0.107223 2024-09-22T03:40:12.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 290 16 43250 43234 0.5823 0.488281 0.0387899 0.107283 2024-09-22T03:40:13.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 291 16 43373 43357 0.58195 0.480469 0.0818016 0.107359 2024-09-22T03:40:14.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 292 16 43477 43461 0.581348 0.40625 0.287737 0.107454 2024-09-22T03:40:15.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 293 16 43582 43566 0.580764 0.410156 0.120808 0.107547 2024-09-22T03:40:16.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 294 16 43700 43684 0.580356 0.460938 0.0933369 0.107648 2024-09-22T03:40:17.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 295 16 43810 43794 0.579845 0.429688 0.141836 0.107726 2024-09-22T03:40:18.766 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 296 16 43919 43903 0.579325 0.425781 0.587098 0.107827 2024-09-22T03:40:19.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 297 16 44041 44025 0.578978 0.476562 0.0539794 0.107903 2024-09-22T03:40:20.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 298 16 44150 44134 0.578464 0.425781 0.0456848 0.107985 2024-09-22T03:40:21.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 299 16 44251 44235 0.577849 0.394531 0.625703 0.108106 2024-09-22T03:40:22.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:2024-09-22T03:40:22.767041+0000 min lat: 0.00649382 max lat: 0.864719 avg lat: 0.108179 2024-09-22T03:40:22.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-22T03:40:22.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout: 300 16 44370 44354 0.577472 0.464844 0.242342 0.108179 2024-09-22T03:40:23.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Total time run: 300.121 2024-09-22T03:40:23.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Total writes made: 44370 2024-09-22T03:40:23.767 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Write size: 4096 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Object size: 4096 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Bandwidth (MB/sec): 0.577502 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Stddev Bandwidth: 0.128039 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Max bandwidth (MB/sec): 1.03906 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Min bandwidth (MB/sec): 0.394531 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Average IOPS: 147 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Stddev IOPS: 32.778 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Max IOPS: 266 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Min IOPS: 101 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Average Latency(s): 0.108202 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Stddev Latency(s): 0.109314 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Max latency(s): 0.864719 2024-09-22T03:40:23.768 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Min latency(s): 0.00649382 2024-09-22T03:40:23.792 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-09-22T03:47:00.161 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Removed 44370 objects 2024-09-22T03:47:00.161 INFO:tasks.radosbench.radosbench.0.smithi122.stdout:Clean up completed and total clean up time :396.368 2024-09-22T03:47:04.324 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-22T03:47:04.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-22T03:47:04.451 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:47:04.443+0000 7f139f66c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:47:04.453 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:47:04.453+0000 7f139f66c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:47:04.842 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'unique_pool_0' removed 2024-09-22T03:47:04.863 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T03:47:04.872 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-22T03:47:04.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:47:04.989+0000 7f10bd063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:47:04.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-09-22T03:47:04.999+0000 7f10bd063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:47:05.212 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T03:47:05.212 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T03:47:05.212 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:47:05.212 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:47:07.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T03:47:11.314 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:47:11.314 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:47:11.314 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:47:13.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:47:13.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T03:47:17.417 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:47:17.418 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:47:17.418 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:47:18.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:47:18.436 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:47:23.519 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:47:23.520 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T03:47:23.520 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-22T03:47:23.520 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:47:23.553 INFO:tasks.ceph.mgr.x:Stopped 2024-09-22T03:47:23.553 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T03:47:23.554 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-22T03:47:23.554 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:47:23.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-09-22T03:47:23.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-09-22T03:47:23.606 INFO:tasks.ceph.mon.a:Stopped 2024-09-22T03:47:23.606 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T03:47:23.606 DEBUG:teuthology.orchestra.run.smithi122:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-22T03:47:23.678 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:47:23.678 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-22T03:47:23.860 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:47:23.860 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-22T03:47:23.934 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:47:23.934 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-22T03:47:23.992 INFO:tasks.ceph:Archiving mon data... 2024-09-22T03:47:23.993 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914779/data/mon.a.tgz 2024-09-22T03:47:23.993 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-09-22T03:47:24.012 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.ij56Uc66op 2024-09-22T03:47:24.013 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ij56Uc66op 2024-09-22T03:47:24.125 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.ij56Uc66op 2024-09-22T03:47:24.210 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.ij56Uc66op is 181KB 2024-09-22T03:47:24.236 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.ij56Uc66op 2024-09-22T03:47:24.251 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T03:47:24.251 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:47:24.371 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:47:24.371 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T03:47:24.373 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914779/remote/smithi122/crash 2024-09-22T03:47:24.374 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:47:24.435 INFO:tasks.ceph:Compressing logs... 2024-09-22T03:47:24.436 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:47:24.511 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-22T03:47:24.511 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28529.log 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28170.log 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28529.log.gz 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29110.log 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28170.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27126.log 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28170.log.gz 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28858.log 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29110.log.gz 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.tmp-client.admin.27126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29278.log 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27126.log.gz 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28082.log 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.28858.log.gz 2024-09-22T03:47:24.512 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29278.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29278.log.gz 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28763.log 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28082.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28082.log.gz 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28707.log 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28763.log: 85.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29023.log 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.28763.log.gz 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29024.log 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28707.log.gz 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29362.log 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29023.log.gz 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.29430.log 2024-09-22T03:47:24.513 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29024.log.gz 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29362.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29362.log.gz 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.0.29430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29407.log 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.0.29430.log.gz 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29236.log 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29407.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29407.log.gz 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28364.log 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29022.log 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29236.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29236.log.gz 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28364.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28364.log.gz 2024-09-22T03:47:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28942.log 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29022.log.gz 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.2.log: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28788.log 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28942.log.gz 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28320.log 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.28788.log.gz 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28557.log 2024-09-22T03:47:24.515 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29149.log 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr: 87.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28320.log.gz 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29150.log 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28557.log.gz 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29547.log 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29149.log.gz 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29320.log 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29150.log.gz 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29547.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29547.log.gz 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28900.log 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29507.log 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29320.log.gz 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28280.log 2024-09-22T03:47:24.516 INFO:teuthology.orchestra.run.smithi122.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28900.log.gz 2024-09-22T03:47:24.517 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.29507.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29507.log.gz 2024-09-22T03:47:24.517 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.28280.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28280.log.gz 2024-09-22T03:47:24.517 INFO:teuthology.orchestra.run.smithi122.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-22T03:47:24.538 INFO:teuthology.orchestra.run.smithi122.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-22T03:47:24.540 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-09-22T03:47:24.540 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m0.049s 2024-09-22T03:47:24.540 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m0.062s 2024-09-22T03:47:24.540 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m0.030s 2024-09-22T03:47:24.540 INFO:tasks.ceph:Archiving logs... 2024-09-22T03:47:24.540 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914779/remote/smithi122/log 2024-09-22T03:47:24.541 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:47:24.649 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T03:47:24.661 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T03:47:24.661 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:47:24.710 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T03:47:24.722 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T03:47:24.722 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:47:24.739 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:47:24.743 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:47:24.743 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-09-22T03:47:24.743 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 -328us[ -335us] +/- 64ms 2024-09-22T03:47:24.743 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 35 +682us[ +682us] +/- 58ms 2024-09-22T03:47:24.743 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -460us[ -467us] +/- 73ms 2024-09-22T03:47:24.743 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:47:24.744 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T03:47:24.758 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T03:47:24.758 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T03:47:24.773 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:47:24.845 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:47:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:47:24.871 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T03:47:24.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T03:47:24.958 INFO:teuthology.task.internal:Duration was 1170.841132 seconds 2024-09-22T03:47:24.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T03:47:24.968 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T03:47:24.968 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:47:25.007 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:47:25.468 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T03:47:25.469 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-09-22T03:47:25.469 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:47:25.529 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T03:47:25.529 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:47:25.596 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T03:47:25.596 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:47:25.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T03:47:25.745 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T03:47:25.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:47:25.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T03:47:25.780 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:47:25.833 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-09-22T03:47:25.854 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:47:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:47:25.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T03:47:25.920 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T03:47:25.921 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914779/remote/smithi122 2024-09-22T03:47:25.921 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:47:25.982 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T03:47:25.982 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:47:26.031 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T03:47:26.048 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T03:47:26.048 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T03:47:26.062 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T03:47:26.063 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:47:26.086 INFO:teuthology.orchestra.run.smithi122.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:47 /home/ubuntu/cephtest 2024-09-22T03:47:26.087 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T03:47:26.101 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T03:47:26.138 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_omap_write} duration: 1170.8411321640015 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-22T03:47:26.139 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:47:26.219 INFO:teuthology.run:pass