2024-09-11T14:24:24.892 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T14:24:24.908 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:24:24.995 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901361 branch: main description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format} email: null first_in_suite: false job_id: '7901361' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(POOL_FULL\) sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a workunit: branch: wip-matanb-crimson-seastore-logs env: CRIMSON_COMPAT: '1' sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5813 sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-seastore-logs suite_path: /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa suite_relpath: qa suite_repo: https://github.com/matan-b/ceph.git suite_sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEUolyCmVKyS4nDkm1L2008caDo2YChaf83PiHAnb8Hb4K4Qhiz1NQ2C8C/RcnwmoSsV6YGcLNFrmW39DHnyaNw= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-11_13:56:22 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T14:24:24.995 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa; will attempt to use it 2024-09-11T14:24:24.996 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks 2024-09-11T14:24:24.996 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T14:24:24.998 INFO:teuthology.task.internal:Checking packages... 2024-09-11T14:24:25.018 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '0d61375b8ab53be4c233f5b93e463edfdff9316a' 2024-09-11T14:24:25.018 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T14:24:25.019 INFO:teuthology.packaging:ref: None 2024-09-11T14:24:25.019 INFO:teuthology.packaging:tag: None 2024-09-11T14:24:25.019 INFO:teuthology.packaging:branch: main 2024-09-11T14:24:25.019 INFO:teuthology.packaging:sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:24:25.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-11T14:24:25.257 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4872.g0d61375b 2024-09-11T14:24:25.259 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T14:24:25.266 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T14:24:25.266 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T14:24:25.274 INFO:teuthology.task.internal:Saving configuration 2024-09-11T14:24:25.285 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T14:24:25.291 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T14:24:25.316 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901361', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 14:19:49.854301', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEUolyCmVKyS4nDkm1L2008caDo2YChaf83PiHAnb8Hb4K4Qhiz1NQ2C8C/RcnwmoSsV6YGcLNFrmW39DHnyaNw='} 2024-09-11T14:24:25.316 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T14:24:25.322 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-11T14:24:25.322 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T14:24:25.356 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f39fa707ac0>, signals=[15]) 2024-09-11T14:24:25.356 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T14:24:25.362 INFO:teuthology.task.internal:Opening connections... 2024-09-11T14:24:25.362 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:24:25.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:24:25.441 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T14:24:25.446 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-09-11T14:24:25.523 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-09-11T14:24:25.523 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T14:24:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-09-11T14:24:25.582 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T14:24:25.582 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T14:24:25.582 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T14:24:25.582 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-09-11T14:24:25.819 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T14:24:25.827 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T14:24:25.834 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T14:24:25.834 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-09-11T14:24:25.851 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T14:24:25.857 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T14:24:25.857 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-09-11T14:24:25.920 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T14:24:25.946 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T14:24:25.960 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-09-11T14:24:25.960 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-09-11T14:24:25.960 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T14:24:25.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:24:25.977 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-09-11T14:24:26.035 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-503.el9.x86_64 2024-09-11T14:24:26.035 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-503.el9.x86_64 2024-09-11T14:24:26.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-09-11T14:24:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.2 MB 00:02 2024-09-11T14:24:48.873 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 20 MB 00:17 2024-09-11T14:24:53.299 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-11T14:24:55.016 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 19 kB 00:00 2024-09-11T14:24:55.879 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-11T14:25:01.526 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-11T14:25:03.758 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T14:25:03.758 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T14:25:03.845 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:25:03.855 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:25:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-09-11T14:25:03.855 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:25:03.855 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:25:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T14:25:03.855 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:25:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T14:25:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T14:25:03.856 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T14:25:03.856 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:25:03.856 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:25:03.856 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:25:03.856 INFO:teuthology.orchestra.run.smithi040.stdout:Install 4 Packages 2024-09-11T14:25:03.856 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:25:03.856 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 79 M 2024-09-11T14:25:03.857 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 126 M 2024-09-11T14:25:03.857 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:25:03.986 INFO:teuthology.orchestra.run.smithi040.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-09-11T14:25:04.695 INFO:teuthology.orchestra.run.smithi040.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 40 MB/s | 29 MB 00:00 2024-09-11T14:25:05.103 INFO:teuthology.orchestra.run.smithi040.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 28 MB/s | 35 MB 00:01 2024-09-11T14:25:05.262 INFO:teuthology.orchestra.run.smithi040.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-11T14:25:05.262 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:25:05.262 INFO:teuthology.orchestra.run.smithi040.stdout:Total 56 MB/s | 79 MB 00:01 2024-09-11T14:25:05.971 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:25:06.360 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:25:06.360 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:25:07.343 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:25:07.343 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:25:09.913 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:25:10.563 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:25:10.662 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:25:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:25:13.573 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:25:18.695 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:25:18.734 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:25:23.861 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:26:03.052 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:26:03.093 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:26:03.465 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:26:03.465 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:26:03.465 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:26:03.465 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:26:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:26:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:26:04.299 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:26:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T14:26:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T14:26:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T14:26:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T14:26:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:26:04.300 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:26:04.628 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:26:04.766 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:26:04.766 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:26:04.766 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T14:26:04.766 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T14:26:04.766 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T14:26:04.767 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T14:26:04.767 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T14:26:04.767 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:26:04.825 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:26:04.826 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:26:04.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2024-09-11T14:26:04.863 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2024-09-11T14:26:04.863 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2024-09-11T14:26:04.863 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.06 2024-09-11T14:26:04.863 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 82.el9 2024-09-11T14:26:04.863 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Group : Unspecified 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 8274670 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : builder@centos.org 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2024-09-11T14:26:04.864 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:26:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T14:26:04.866 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T14:26:04.866 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 5.14.0-505.el9.x86_64 2024-09-11T14:26:04.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T14:26:05.541 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-09-11T14:26:06.934 INFO:teuthology.orchestra.run.smithi040.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T14:26:06.965 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-09-11T14:26:06.969 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-09-11T14:26:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.ujGBED17IR 2024-09-11T14:26:06.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ujGBED17IR 2024-09-11T14:26:07.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.ujGBED17IR 2024-09-11T14:26:07.188 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.ujGBED17IR is 6KB 2024-09-11T14:26:07.237 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.ujGBED17IR 2024-09-11T14:26:07.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T14:26:07.313 INFO:teuthology.orchestra.run.smithi040.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T14:26:07.314 INFO:teuthology.orchestra.run.smithi040.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T14:26:07.314 INFO:teuthology.orchestra.run.smithi040.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T14:26:07.314 INFO:teuthology.orchestra.run.smithi040.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T14:26:07.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T14:26:07.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-09-11T14:26:37.489 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T14:26:37.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-11T14:26:37.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:27:37.554 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T14:27:46.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-11T14:27:46.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:27:50.051 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-09-11T14:27:50.483 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-11T14:27:50.483 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T14:27:50.484 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-09-11T14:27:50.499 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-505.el9.x86_64 2024-09-11T14:27:50.499 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T14:27:50.499 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T14:27:50.499 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T14:27:51.501 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T14:27:51.501 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T14:27:51.584 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-09-11T14:27:51.604 DEBUG:teuthology.parallel:result is None 2024-09-11T14:27:51.604 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T14:27:51.611 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T14:27:51.611 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T14:27:51.627 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T14:27:51.635 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T14:27:51.645 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T14:27:51.645 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T14:27:51.691 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T14:27:51.698 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T14:27:51.699 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T14:27:51.760 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:27:51.770 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:27:51.772 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T14:27:51.779 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T14:27:51.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T14:27:51.839 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T14:27:51.859 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T14:27:51.860 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T14:27:51.895 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:27:51.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:27:52.036 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:27:52.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:27:52.162 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:27:52.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T14:27:52.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-09-11T14:27:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:27:52.420 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T14:27:52.429 INFO:teuthology.task.internal:Starting timer... 2024-09-11T14:27:52.429 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T14:27:52.491 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T14:27:52.501 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-09-11T14:27:52.569 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-09-11T14:27:52.571 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T14:27:52.571 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-09-11T14:27:52.590 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-09-11T14:27:52.590 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive'} 2024-09-11T14:27:52.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:27:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:27:52.654 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T14:27:52.654 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-09-11T14:27:52.731 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T14:27:52.741 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T14:27:52.753 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T14:27:52.754 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T14:31:40.432 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com')] 2024-09-11T14:31:40.433 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-11T14:31:40.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:31:40.517 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-09-11T14:31:40.597 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-11T14:31:40.597 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T14:31:40.607 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T14:31:40.608 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T14:31:40.608 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:31:40.668 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T14:31:40.683 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T14:31:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-09-11T14:31:40.721 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-09-11T14:31:40.736 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T14:31:40.750 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T14:31:40.810 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:31:40.813 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:31:40.814 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-09-11T14:31:40.814 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:31:40.814 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:31:40.814 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:31:40.814 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:31:40.815 INFO:teuthology.run_tasks:Running task install... 2024-09-11T14:31:40.823 DEBUG:teuthology.task.install:project ceph 2024-09-11T14:31:40.824 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}} 2024-09-11T14:31:40.824 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'} 2024-09-11T14:31:40.824 INFO:teuthology.task.install:Using flavor: crimson 2024-09-11T14:31:40.829 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T14:31:40.829 INFO:teuthology.task.install:extra packages: [] 2024-09-11T14:31:40.830 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'tag': None, 'wait_for_package': False} 2024-09-11T14:31:40.830 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:31:40.980 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/ 2024-09-11T14:31:40.980 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4872.g0d61375b 2024-09-11T14:31:41.093 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T14:31:41.093 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:31:41.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T14:31:41.120 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T14:31:41.121 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0d61375b8ab53be4c233f5b93e463edfdff9316a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T14:31:41.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T14:31:41.264 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T14:31:41.328 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-09-11T14:31:41.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-09-11T14:31:41.567 INFO:teuthology.orchestra.run.smithi040.stdout:45 files removed 2024-09-11T14:31:41.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw 2024-09-11T14:31:42.165 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 327 kB/s | 89 kB 00:00 2024-09-11T14:31:42.458 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-11T14:31:42.741 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 8.5 kB/s | 2.1 kB 00:00 2024-09-11T14:31:45.083 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.2 MB 00:02 2024-09-11T14:31:54.184 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-11T14:32:00.893 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-11T14:32:02.585 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 19 kB 00:00 2024-09-11T14:32:03.381 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-11T14:32:09.162 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-11T14:32:11.600 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:32:11.603 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 10 M 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.8 M 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.1 M 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 20 M 2024-09-11T14:32:11.604 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 25 k 2024-09-11T14:32:11.605 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T14:32:11.605 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T14:32:11.605 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T14:32:11.605 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 808 k 2024-09-11T14:32:11.605 INFO:teuthology.orchestra.run.smithi040.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T14:32:11.605 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T14:32:11.605 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T14:32:11.605 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T14:32:11.606 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 558 k 2024-09-11T14:32:11.606 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T14:32:11.606 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.4 M 2024-09-11T14:32:11.606 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T14:32:11.606 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T14:32:11.606 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T14:32:11.607 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 45 k 2024-09-11T14:32:11.607 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 143 k 2024-09-11T14:32:11.607 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 159 k 2024-09-11T14:32:11.607 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T14:32:11.607 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 316 k 2024-09-11T14:32:11.608 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 303 k 2024-09-11T14:32:11.608 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 99 k 2024-09-11T14:32:11.608 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T14:32:11.608 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T14:32:11.608 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T14:32:11.608 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T14:32:11.608 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-09-11T14:32:11.609 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T14:32:11.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:32:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:32:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Install 31 Packages 2024-09-11T14:32:11.609 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 2 Packages 2024-09-11T14:32:11.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 60 M 2024-09-11T14:32:11.610 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:32:12.205 INFO:teuthology.orchestra.run.smithi040.stdout:(1/33): ceph-base-19.3.0-4872.g0d61375b.el9.x86 12 MB/s | 6.1 MB 00:00 2024-09-11T14:32:12.255 INFO:teuthology.orchestra.run.smithi040.stdout:(2/33): ceph-selinux-19.3.0-4872.g0d61375b.el9. 503 kB/s | 25 kB 00:00 2024-09-11T14:32:12.355 INFO:teuthology.orchestra.run.smithi040.stdout:(3/33): libcephfs2-19.3.0-4872.g0d61375b.el9.x8 7.9 MB/s | 808 kB 00:00 2024-09-11T14:32:12.447 INFO:teuthology.orchestra.run.smithi040.stdout:(4/33): libradosstriper1-19.3.0-4872.g0d61375b. 6.0 MB/s | 558 kB 00:00 2024-09-11T14:32:12.614 INFO:teuthology.orchestra.run.smithi040.stdout:(5/33): ceph-radosgw-19.3.0-4872.g0d61375b.el9. 11 MB/s | 10 MB 00:00 2024-09-11T14:32:12.823 INFO:teuthology.orchestra.run.smithi040.stdout:(6/33): ceph-common-19.3.0-4872.g0d61375b.el9.x 17 MB/s | 20 MB 00:01 2024-09-11T14:32:12.839 INFO:teuthology.orchestra.run.smithi040.stdout:(7/33): python3-ceph-argparse-19.3.0-4872.g0d61 200 kB/s | 45 kB 00:00 2024-09-11T14:32:12.873 INFO:teuthology.orchestra.run.smithi040.stdout:(8/33): python3-ceph-common-19.3.0-4872.g0d6137 2.8 MB/s | 143 kB 00:00 2024-09-11T14:32:12.898 INFO:teuthology.orchestra.run.smithi040.stdout:(9/33): python3-cephfs-19.3.0-4872.g0d61375b.el 2.6 MB/s | 159 kB 00:00 2024-09-11T14:32:12.982 INFO:teuthology.orchestra.run.smithi040.stdout:(10/33): librgw2-19.3.0-4872.g0d61375b.el9.x86_ 10 MB/s | 5.4 MB 00:00 2024-09-11T14:32:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:(11/33): python3-rados-19.3.0-4872.g0d61375b.el 2.3 MB/s | 316 kB 00:00 2024-09-11T14:32:13.033 INFO:teuthology.orchestra.run.smithi040.stdout:(12/33): python3-rgw-19.3.0-4872.g0d61375b.el9. 1.9 MB/s | 99 kB 00:00 2024-09-11T14:32:13.058 INFO:teuthology.orchestra.run.smithi040.stdout:(13/33): python3-rbd-19.3.0-4872.g0d61375b.el9. 1.9 MB/s | 303 kB 00:00 2024-09-11T14:32:13.458 INFO:teuthology.orchestra.run.smithi040.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 237 kB/s | 107 kB 00:00 2024-09-11T14:32:13.517 INFO:teuthology.orchestra.run.smithi040.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 349 kB/s | 160 kB 00:00 2024-09-11T14:32:13.558 INFO:teuthology.orchestra.run.smithi040.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-09-11T14:32:13.586 INFO:teuthology.orchestra.run.smithi040.stdout:(17/33): libnbd-1.20.2-2.el9.x86_64.rpm 308 kB/s | 171 kB 00:00 2024-09-11T14:32:13.703 INFO:teuthology.orchestra.run.smithi040.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-09-11T14:32:13.845 INFO:teuthology.orchestra.run.smithi040.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-09-11T14:32:13.895 INFO:teuthology.orchestra.run.smithi040.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 1.7 MB/s | 662 kB 00:00 2024-09-11T14:32:13.954 INFO:teuthology.orchestra.run.smithi040.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 330 kB/s | 36 kB 00:00 2024-09-11T14:32:14.004 INFO:teuthology.orchestra.run.smithi040.stdout:(22/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.0 MB/s | 309 kB 00:00 2024-09-11T14:32:14.046 INFO:teuthology.orchestra.run.smithi040.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 644 kB/s | 221 kB 00:00 2024-09-11T14:32:14.071 INFO:teuthology.orchestra.run.smithi040.stdout:(24/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 270 kB/s | 47 kB 00:00 2024-09-11T14:32:14.113 INFO:teuthology.orchestra.run.smithi040.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-09-11T14:32:14.138 INFO:teuthology.orchestra.run.smithi040.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-09-11T14:32:14.164 INFO:teuthology.orchestra.run.smithi040.stdout:(27/33): liboath-2.6.7-2.el9.x86_64.rpm 578 kB/s | 49 kB 00:00 2024-09-11T14:32:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-11T14:32:14.239 INFO:teuthology.orchestra.run.smithi040.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-11T14:32:14.306 INFO:teuthology.orchestra.run.smithi040.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-11T14:32:14.390 INFO:teuthology.orchestra.run.smithi040.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-11T14:32:14.574 INFO:teuthology.orchestra.run.smithi040.stdout:(32/33): librados2-19.3.0-4872.g0d61375b.el9.x8 10 MB/s | 3.5 MB 00:00 2024-09-11T14:32:14.682 INFO:teuthology.orchestra.run.smithi040.stdout:(33/33): librbd1-19.3.0-4872.g0d61375b.el9.x86_ 10 MB/s | 3.8 MB 00:00 2024-09-11T14:32:14.688 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:32:14.688 INFO:teuthology.orchestra.run.smithi040.stdout:Total 20 MB/s | 60 MB 00:03 2024-09-11T14:32:14.808 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:32:14.850 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:32:14.850 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:32:15.542 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:32:15.543 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:32:17.251 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:32:17.785 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-11T14:32:17.867 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:32:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:32:18.052 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:32:18.077 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:32:18.140 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:32:18.187 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-11T14:32:18.215 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-11T14:32:18.241 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-11T14:32:18.263 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:32:18.309 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:32:18.532 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:32:18.564 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:32:18.599 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-11T14:32:18.660 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 11/35 2024-09-11T14:32:18.688 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:32:18.779 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-11T14:32:18.801 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:32:18.890 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:32:18.934 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-11T14:32:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-11T14:32:19.088 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-11T14:32:19.138 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-11T14:32:19.175 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 19/35 2024-09-11T14:32:19.204 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-11T14:32:19.257 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-11T14:32:19.660 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-11T14:32:19.716 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:32:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:32:19.834 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 24/35 2024-09-11T14:32:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-11T14:32:20.501 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-11T14:32:20.527 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:32:20.599 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:32:20.680 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 28/35 2024-09-11T14:32:20.713 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 29/35 2024-09-11T14:32:23.298 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:32:23.334 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:32:24.061 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:32:24.096 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:32:24.196 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:32:24.196 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T14:32:24.196 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T14:32:24.196 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:24.239 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-09-11T14:32:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:36.366 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:32:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:32:36.393 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:32:36.393 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T14:32:36.393 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:32:36.393 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:32:36.393 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:36.395 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:32:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:32:36.485 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 1/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 5/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 6/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 7/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 8/35 2024-09-11T14:32:37.967 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 9/35 2024-09-11T14:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 10/35 2024-09-11T14:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 11/35 2024-09-11T14:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 13/35 2024-09-11T14:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-11T14:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-11T14:32:37.968 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-11T14:32:37.969 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-11T14:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-11T14:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-11T14:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-11T14:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-11T14:32:37.970 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 34/35 2024-09-11T14:32:39.063 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:32:39.063 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:39.063 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-09-11T14:32:39.063 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.064 INFO:teuthology.orchestra.run.smithi040.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.065 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:39.066 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:32:39.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-test 2024-09-11T14:32:40.064 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:31 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:32:40.202 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:32:40.203 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:32:40.203 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:32:40.203 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:32:40.203 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 85 M 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 175 k 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:32:40.204 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:32:40.205 INFO:teuthology.orchestra.run.smithi040.stdout:Install 9 Packages 2024-09-11T14:32:40.205 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:32:40.205 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 87 M 2024-09-11T14:32:40.205 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 797 M 2024-09-11T14:32:40.205 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:32:40.924 INFO:teuthology.orchestra.run.smithi040.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 518 kB/s | 94 kB 00:00 2024-09-11T14:32:40.949 INFO:teuthology.orchestra.run.smithi040.stdout:(2/9): libcephsqlite-19.3.0-4872.g0d61375b.el9. 844 kB/s | 175 kB 00:00 2024-09-11T14:32:41.491 INFO:teuthology.orchestra.run.smithi040.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 337 kB/s | 183 kB 00:00 2024-09-11T14:32:41.650 INFO:teuthology.orchestra.run.smithi040.stdout:(4/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 708 kB/s | 513 kB 00:00 2024-09-11T14:32:41.852 INFO:teuthology.orchestra.run.smithi040.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-11T14:32:41.883 INFO:teuthology.orchestra.run.smithi040.stdout:(6/9): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-11T14:32:41.925 INFO:teuthology.orchestra.run.smithi040.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.9 MB/s | 122 kB 00:00 2024-09-11T14:32:41.951 INFO:teuthology.orchestra.run.smithi040.stdout:(8/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 634 kB/s | 64 kB 00:00 2024-09-11T14:32:43.359 INFO:teuthology.orchestra.run.smithi040.stdout:(9/9): ceph-test-19.3.0-4872.g0d61375b.el9.x86_ 32 MB/s | 85 MB 00:02 2024-09-11T14:32:43.362 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:32:43.362 INFO:teuthology.orchestra.run.smithi040.stdout:Total 28 MB/s | 87 MB 00:03 2024-09-11T14:32:43.448 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:32:43.475 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:32:43.475 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:32:44.107 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:32:44.107 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:32:45.020 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:32:45.093 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-11T14:32:45.151 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-11T14:32:45.198 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-11T14:32:45.231 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-11T14:32:45.280 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:32:45.304 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:32:45.376 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:32:45.422 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-11T14:32:45.448 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:32:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:32:57.474 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 1/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-11T14:33:03.575 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-11T14:33:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-11T14:33:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:03.973 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:33:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:03.974 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:33:04.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph 2024-09-11T14:33:04.898 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:33:05.041 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:33:05.044 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:05.044 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:33:05.044 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:05.044 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:33:05.045 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.5 k 2024-09-11T14:33:05.045 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:33:05.045 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 2.2 M 2024-09-11T14:33:05.045 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 1.6 M 2024-09-11T14:33:05.045 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 291 k 2024-09-11T14:33:05.045 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.1 M 2024-09-11T14:33:05.045 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 71 M 2024-09-11T14:33:05.045 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T14:33:05.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T14:33:05.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout:Install 37 Packages 2024-09-11T14:33:05.048 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:05.049 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 84 M 2024-09-11T14:33:05.049 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 896 M 2024-09-11T14:33:05.049 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:33:05.573 INFO:teuthology.orchestra.run.smithi040.stdout:(1/37): ceph-19.3.0-4872.g0d61375b.el9.x86_64.r 47 kB/s | 6.5 kB 00:00 2024-09-11T14:33:05.757 INFO:teuthology.orchestra.run.smithi040.stdout:(2/37): ceph-mgr-19.3.0-4872.g0d61375b.el9.x86_ 4.9 MB/s | 1.6 MB 00:00 2024-09-11T14:33:05.882 INFO:teuthology.orchestra.run.smithi040.stdout:(3/37): ceph-mds-19.3.0-4872.g0d61375b.el9.x86_ 4.9 MB/s | 2.2 MB 00:00 2024-09-11T14:33:05.957 INFO:teuthology.orchestra.run.smithi040.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4872.g0d61 3.8 MB/s | 291 kB 00:00 2024-09-11T14:33:06.158 INFO:teuthology.orchestra.run.smithi040.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-11T14:33:06.316 INFO:teuthology.orchestra.run.smithi040.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 7.8 MB/s | 1.2 MB 00:00 2024-09-11T14:33:06.400 INFO:teuthology.orchestra.run.smithi040.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-09-11T14:33:06.508 INFO:teuthology.orchestra.run.smithi040.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-09-11T14:33:06.592 INFO:teuthology.orchestra.run.smithi040.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.5 MB/s | 125 kB 00:00 2024-09-11T14:33:06.734 INFO:teuthology.orchestra.run.smithi040.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-09-11T14:33:06.918 INFO:teuthology.orchestra.run.smithi040.stdout:(11/37): ceph-mon-19.3.0-4872.g0d61375b.el9.x86 3.8 MB/s | 5.1 MB 00:01 2024-09-11T14:33:07.160 INFO:teuthology.orchestra.run.smithi040.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 443 kB/s | 188 kB 00:00 2024-09-11T14:33:07.418 INFO:teuthology.orchestra.run.smithi040.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 344 kB/s | 172 kB 00:00 2024-09-11T14:33:07.435 INFO:teuthology.orchestra.run.smithi040.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 126 kB/s | 35 kB 00:00 2024-09-11T14:33:07.785 INFO:teuthology.orchestra.run.smithi040.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 146 kB/s | 51 kB 00:00 2024-09-11T14:33:07.835 INFO:teuthology.orchestra.run.smithi040.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 185 kB/s | 77 kB 00:00 2024-09-11T14:33:07.920 INFO:teuthology.orchestra.run.smithi040.stdout:(17/37): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-09-11T14:33:07.961 INFO:teuthology.orchestra.run.smithi040.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-11T14:33:07.987 INFO:teuthology.orchestra.run.smithi040.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-11T14:33:08.012 INFO:teuthology.orchestra.run.smithi040.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.8 MB/s | 172 kB 00:00 2024-09-11T14:33:08.046 INFO:teuthology.orchestra.run.smithi040.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-11T14:33:08.071 INFO:teuthology.orchestra.run.smithi040.stdout:(22/37): python3-toml-0.10.2-6.el9.noarch.rpm 146 kB/s | 42 kB 00:00 2024-09-11T14:33:08.096 INFO:teuthology.orchestra.run.smithi040.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-11T14:33:08.130 INFO:teuthology.orchestra.run.smithi040.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 942 kB/s | 23 kB 00:00 2024-09-11T14:33:08.155 INFO:teuthology.orchestra.run.smithi040.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 851 kB/s | 19 kB 00:00 2024-09-11T14:33:08.180 INFO:teuthology.orchestra.run.smithi040.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-09-11T14:33:08.205 INFO:teuthology.orchestra.run.smithi040.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-11T14:33:08.231 INFO:teuthology.orchestra.run.smithi040.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 928 kB/s | 46 kB 00:00 2024-09-11T14:33:08.248 INFO:teuthology.orchestra.run.smithi040.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-11T14:33:08.273 INFO:teuthology.orchestra.run.smithi040.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 668 kB/s | 16 kB 00:00 2024-09-11T14:33:08.298 INFO:teuthology.orchestra.run.smithi040.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.6 MB/s | 90 kB 00:00 2024-09-11T14:33:08.324 INFO:teuthology.orchestra.run.smithi040.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-09-11T14:33:08.349 INFO:teuthology.orchestra.run.smithi040.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-11T14:33:08.382 INFO:teuthology.orchestra.run.smithi040.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-09-11T14:33:08.416 INFO:teuthology.orchestra.run.smithi040.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 595 kB/s | 20 kB 00:00 2024-09-11T14:33:08.442 INFO:teuthology.orchestra.run.smithi040.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-11T14:33:09.309 INFO:teuthology.orchestra.run.smithi040.stdout:(37/37): ceph-osd-19.3.0-4872.g0d61375b.el9.x86 20 MB/s | 71 MB 00:03 2024-09-11T14:33:09.314 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:33:09.314 INFO:teuthology.orchestra.run.smithi040.stdout:Total 20 MB/s | 84 MB 00:04 2024-09-11T14:33:09.470 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:33:09.516 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:33:09.516 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:33:10.379 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:33:10.379 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:33:12.098 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:33:12.153 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T14:33:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T14:33:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T14:33:12.294 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T14:33:12.333 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T14:33:12.366 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T14:33:12.472 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T14:33:12.564 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T14:33:12.611 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T14:33:12.645 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T14:33:12.675 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T14:33:12.709 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T14:33:12.742 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T14:33:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T14:33:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T14:33:12.861 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T14:33:12.910 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T14:33:12.950 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T14:33:13.027 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T14:33:13.181 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T14:33:13.262 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T14:33:13.320 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T14:33:13.358 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T14:33:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T14:33:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T14:33:13.676 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T14:33:13.729 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T14:33:13.811 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T14:33:13.917 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T14:33:13.954 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T14:33:14.089 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T14:33:14.292 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 32/37 2024-09-11T14:33:14.322 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:33:14.350 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:33:14.350 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:33:14.350 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T14:33:14.350 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:33:14.350 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:33:14.350 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:25.581 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:33:25.609 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:33:25.609 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:33:25.609 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T14:33:25.609 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:33:25.609 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:33:25.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:26.189 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:33:26.215 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:33:26.215 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:33:26.215 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T14:33:26.215 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:33:26.215 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:33:26.215 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:26.457 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:33:26.481 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:33:26.482 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:33:26.482 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T14:33:26.482 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:33:26.482 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:33:26.482 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:33:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 1/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 3/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 4/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 5/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 6/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T14:33:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T14:33:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T14:33:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T14:33:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T14:33:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T14:33:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T14:33:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T14:33:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T14:33:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T14:33:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T14:33:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T14:33:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T14:33:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T14:33:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T14:33:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T14:33:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T14:33:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T14:33:34.141 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T14:33:34.141 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T14:33:34.142 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T14:33:34.143 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:34.144 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:33:34.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base 2024-09-11T14:33:34.999 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:25 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:33:35.091 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:33:35.136 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:33:35.137 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:33:35.137 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:33:35.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install cephadm 2024-09-11T14:33:35.786 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:26 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:33:35.922 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:33:35.923 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:33:35.923 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:35.923 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:33:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 784 k 2024-09-11T14:33:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:35.924 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:33:35.924 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:35.924 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-09-11T14:33:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:35.924 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 784 k 2024-09-11T14:33:35.924 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 791 k 2024-09-11T14:33:35.924 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:33:36.916 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm-19.3.0-4872.g0d61375b.el9.noarch.rpm 791 kB/s | 784 kB 00:00 2024-09-11T14:33:36.916 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:33:36.917 INFO:teuthology.orchestra.run.smithi040.stdout:Total 790 kB/s | 784 kB 00:00 2024-09-11T14:33:36.917 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:33:36.924 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:33:36.924 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:33:36.930 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:33:36.931 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:33:37.008 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:33:37.442 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:33:37.540 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:33:37.900 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:33:38.151 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:33:38.151 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:38.152 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:33:38.152 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:33:38.152 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:38.152 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:33:38.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T14:33:38.967 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:29 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:33:39.106 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:33:39.106 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:39.106 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-09-11T14:33:39.106 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:39.107 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:33:39.107 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 158 k 2024-09-11T14:33:39.107 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:39.107 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:33:39.107 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:39.107 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-09-11T14:33:39.107 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:39.108 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 158 k 2024-09-11T14:33:39.108 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 644 k 2024-09-11T14:33:39.108 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:33:39.355 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-immutable-object-cache-19.3.0-4872.g0d6137 640 kB/s | 158 kB 00:00 2024-09-11T14:33:39.355 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:33:39.356 INFO:teuthology.orchestra.run.smithi040.stdout:Total 635 kB/s | 158 kB 00:00 2024-09-11T14:33:39.356 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:33:39.367 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:33:39.367 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:33:39.429 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:33:39.429 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:33:39.590 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:33:39.673 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:33:39.699 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:33:39.699 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:33:39.699 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T14:33:39.699 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:40.630 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:33:40.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:40.630 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:33:40.630 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:33:40.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:40.630 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:33:40.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr 2024-09-11T14:33:41.443 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:32 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:33:41.537 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:33:41.582 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:33:41.583 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:33:41.583 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:33:41.658 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T14:33:42.246 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:33 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:33:42.388 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 14 M 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 34 k 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 16 k 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T14:33:42.389 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout:Install 11 Packages 2024-09-11T14:33:42.390 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:42.391 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 18 M 2024-09-11T14:33:42.391 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 155 M 2024-09-11T14:33:42.391 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:33:42.777 INFO:teuthology.orchestra.run.smithi040.stdout:(1/11): ceph-grafana-dashboards-19.3.0-4872.g0d 233 kB/s | 34 kB 00:00 2024-09-11T14:33:42.803 INFO:teuthology.orchestra.run.smithi040.stdout:(2/11): ceph-prometheus-alerts-19.3.0-4872.g0d6 96 kB/s | 16 kB 00:00 2024-09-11T14:33:43.011 INFO:teuthology.orchestra.run.smithi040.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-09-11T14:33:43.061 INFO:teuthology.orchestra.run.smithi040.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-11T14:33:43.087 INFO:teuthology.orchestra.run.smithi040.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-09-11T14:33:43.120 INFO:teuthology.orchestra.run.smithi040.stdout:(6/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-09-11T14:33:43.171 INFO:teuthology.orchestra.run.smithi040.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-11T14:33:43.196 INFO:teuthology.orchestra.run.smithi040.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-11T14:33:43.229 INFO:teuthology.orchestra.run.smithi040.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2024-09-11T14:33:43.280 INFO:teuthology.orchestra.run.smithi040.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-11T14:33:43.438 INFO:teuthology.orchestra.run.smithi040.stdout:(11/11): ceph-mgr-dashboard-19.3.0-4872.g0d6137 18 MB/s | 14 MB 00:00 2024-09-11T14:33:43.443 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:33:43.443 INFO:teuthology.orchestra.run.smithi040.stdout:Total 17 MB/s | 18 MB 00:01 2024-09-11T14:33:43.475 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:33:43.496 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:33:43.497 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:33:43.772 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:33:43.772 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:33:44.338 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:33:44.421 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-11T14:33:44.458 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-11T14:33:44.563 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-11T14:33:44.699 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-11T14:33:44.746 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-11T14:33:44.833 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-11T14:33:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-11T14:33:44.917 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:33:44.950 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 9/11 2024-09-11T14:33:48.336 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 10/11 2024-09-11T14:33:48.396 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:33:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:33:50.325 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 1/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 2/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 3/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-11T14:33:50.326 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-11T14:33:50.643 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T14:33:50.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T14:33:50.645 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T14:33:50.645 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T14:33:50.645 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T14:33:50.645 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:50.645 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:33:50.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T14:33:51.474 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:33:51.616 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 7.4 M 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T14:33:51.617 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================= 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout:Install 12 Packages 2024-09-11T14:33:51.618 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:33:51.619 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 38 M 2024-09-11T14:33:51.619 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 212 M 2024-09-11T14:33:51.619 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:33:52.045 INFO:teuthology.orchestra.run.smithi040.stdout:(1/12): libgfortran-11.5.0-2.el9.x86_64.rpm 5.3 MB/s | 801 kB 00:00 2024-09-11T14:33:52.070 INFO:teuthology.orchestra.run.smithi040.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-11T14:33:52.403 INFO:teuthology.orchestra.run.smithi040.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 15 MB/s | 7.4 MB 00:00 2024-09-11T14:33:53.445 INFO:teuthology.orchestra.run.smithi040.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 23 kB/s | 33 kB 00:01 2024-09-11T14:33:54.212 INFO:teuthology.orchestra.run.smithi040.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 10 kB/s | 18 kB 00:01 2024-09-11T14:33:54.779 INFO:teuthology.orchestra.run.smithi040.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 29 kB/s | 38 kB 00:01 2024-09-11T14:33:54.846 INFO:teuthology.orchestra.run.smithi040.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.1 MB/s | 3.0 MB 00:02 2024-09-11T14:33:56.771 INFO:teuthology.orchestra.run.smithi040.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 123 kB/s | 245 kB 00:01 2024-09-11T14:33:57.405 INFO:teuthology.orchestra.run.smithi040.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 1.5 MB/s | 4.9 MB 00:03 2024-09-11T14:33:57.888 INFO:teuthology.orchestra.run.smithi040.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.7 MB/s | 5.1 MB 00:03 2024-09-11T14:33:58.822 INFO:teuthology.orchestra.run.smithi040.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 122 kB/s | 250 kB 00:02 2024-09-11T14:34:01.013 INFO:teuthology.orchestra.run.smithi040.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.5 MB/s | 16 MB 00:03 2024-09-11T14:34:01.015 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:01.015 INFO:teuthology.orchestra.run.smithi040.stdout:Total 4.1 MB/s | 38 MB 00:09 2024-09-11T14:34:01.386 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:01.415 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:01.415 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:01.868 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:01.869 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:02.781 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:02.852 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T14:34:02.895 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:34:02.999 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:34:03.046 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T14:34:03.251 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T14:34:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T14:34:03.365 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T14:34:04.054 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T14:34:04.179 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T14:34:05.518 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:34:06.624 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T14:34:06.652 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:34:09.183 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:34:09.183 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 1/12 2024-09-11T14:34:09.183 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:34:09.184 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T14:34:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T14:34:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:09.585 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:34:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T14:34:09.585 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:09.586 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:09.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-rook 2024-09-11T14:34:10.557 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:01 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:10.698 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:10.699 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 49 k 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T14:34:10.700 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout:Install 13 Packages 2024-09-11T14:34:10.701 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:10.702 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.2 M 2024-09-11T14:34:10.702 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 26 M 2024-09-11T14:34:10.702 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:11.196 INFO:teuthology.orchestra.run.smithi040.stdout:(1/13): ceph-mgr-rook-19.3.0-4872.g0d61375b.el9 271 kB/s | 49 kB 00:00 2024-09-11T14:34:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 66 kB/s | 19 kB 00:00 2024-09-11T14:34:11.329 INFO:teuthology.orchestra.run.smithi040.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 81 kB/s | 26 kB 00:00 2024-09-11T14:34:11.680 INFO:teuthology.orchestra.run.smithi040.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-11T14:34:11.763 INFO:teuthology.orchestra.run.smithi040.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 392 kB/s | 222 kB 00:00 2024-09-11T14:34:11.789 INFO:teuthology.orchestra.run.smithi040.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 608 kB/s | 279 kB 00:00 2024-09-11T14:34:11.814 INFO:teuthology.orchestra.run.smithi040.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 401 kB/s | 54 kB 00:00 2024-09-11T14:34:11.873 INFO:teuthology.orchestra.run.smithi040.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 295 kB/s | 32 kB 00:00 2024-09-11T14:34:11.898 INFO:teuthology.orchestra.run.smithi040.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 130 kB/s | 14 kB 00:00 2024-09-11T14:34:11.923 INFO:teuthology.orchestra.run.smithi040.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-11T14:34:11.948 INFO:teuthology.orchestra.run.smithi040.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.8 MB/s | 220 kB 00:00 2024-09-11T14:34:11.973 INFO:teuthology.orchestra.run.smithi040.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-11T14:34:12.007 INFO:teuthology.orchestra.run.smithi040.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-11T14:34:12.010 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:12.011 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-11T14:34:12.082 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:12.099 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:12.099 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:12.200 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:12.200 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:12.865 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:13.029 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T14:34:13.093 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T14:34:13.149 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T14:34:13.185 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T14:34:13.227 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T14:34:13.332 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T14:34:13.551 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T14:34:13.604 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T14:34:14.364 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T14:34:14.417 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T14:34:14.451 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T14:34:14.500 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T14:34:14.528 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 1/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T14:34:15.332 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T14:34:15.333 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T14:34:15.333 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T14:34:15.333 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T14:34:15.333 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T14:34:15.667 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T14:34:15.667 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:15.667 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T14:34:15.668 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T14:34:15.669 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T14:34:15.669 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T14:34:15.669 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:15.669 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:15.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T14:34:16.514 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:07 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:16.654 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:16.654 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:16.654 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:34:16.654 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 154 k 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout:Install 6 Packages 2024-09-11T14:34:16.655 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 7.0 M 2024-09-11T14:34:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 33 M 2024-09-11T14:34:16.656 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:16.977 INFO:teuthology.orchestra.run.smithi040.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4872.g0d61375b.e 724 kB/s | 154 kB 00:00 2024-09-11T14:34:17.027 INFO:teuthology.orchestra.run.smithi040.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-11T14:34:17.052 INFO:teuthology.orchestra.run.smithi040.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-11T14:34:17.077 INFO:teuthology.orchestra.run.smithi040.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-11T14:34:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 389 kB/s | 248 kB 00:00 2024-09-11T14:34:18.386 INFO:teuthology.orchestra.run.smithi040.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.7 MB/s | 6.0 MB 00:01 2024-09-11T14:34:18.388 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:18.388 INFO:teuthology.orchestra.run.smithi040.stdout:Total 4.1 MB/s | 7.0 MB 00:01 2024-09-11T14:34:18.462 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:18.474 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:18.474 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:18.580 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:18.582 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:18.900 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:19.084 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T14:34:19.139 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T14:34:19.701 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T14:34:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T14:34:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T14:34:19.900 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:34:20.653 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:34:20.653 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/6 2024-09-11T14:34:20.653 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T14:34:20.653 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T14:34:20.653 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T14:34:20.653 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T14:34:20.925 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T14:34:20.925 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:20.925 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:20.925 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:34:20.925 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T14:34:20.925 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T14:34:20.926 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T14:34:20.926 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T14:34:20.926 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T14:34:20.926 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:20.926 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:21.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-fuse 2024-09-11T14:34:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:12 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:21.918 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:21.918 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:21.918 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:34:21.918 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:21.918 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:21.918 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 884 k 2024-09-11T14:34:21.918 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-09-11T14:34:21.919 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T14:34:21.919 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:21.919 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:21.919 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:21.919 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2024-09-11T14:34:21.919 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:21.919 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 964 k 2024-09-11T14:34:21.920 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 4.1 M 2024-09-11T14:34:21.920 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:22.171 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 626 kB/s | 80 kB 00:00 2024-09-11T14:34:22.355 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): ceph-fuse-19.3.0-4872.g0d61375b.el9.x86_ 2.8 MB/s | 884 kB 00:00 2024-09-11T14:34:22.355 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:22.356 INFO:teuthology.orchestra.run.smithi040.stdout:Total 2.2 MB/s | 964 kB 00:00 2024-09-11T14:34:22.368 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:22.395 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:22.396 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:22.482 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:22.483 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:22.830 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T14:34:22.899 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:34:23.612 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:34:23.612 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/2 2024-09-11T14:34:23.919 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T14:34:23.919 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:23.919 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:23.919 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T14:34:23.919 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:23.919 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:24.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-volume 2024-09-11T14:34:24.817 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:15 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:24.955 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:24.956 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:24.956 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-09-11T14:34:24.956 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:24.956 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:24.956 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 285 k 2024-09-11T14:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-09-11T14:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 285 k 2024-09-11T14:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.3 M 2024-09-11T14:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:25.193 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-volume-19.3.0-4872.g0d61375b.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-11T14:34:25.193 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:25.194 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-11T14:34:25.194 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:25.203 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:25.204 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:25.234 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:25.234 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:25.464 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:34:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:34:25.546 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:34:25.546 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T14:34:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:34:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:26.524 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:34:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:26.524 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:26.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados-devel 2024-09-11T14:34:27.374 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:18 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:27.515 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:27.515 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:27.515 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 125 k 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:27.516 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 125 k 2024-09-11T14:34:27.517 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 445 k 2024-09-11T14:34:27.517 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:27.698 INFO:teuthology.orchestra.run.smithi040.stdout:librados-devel-19.3.0-4872.g0d61375b.el9.x86_64 688 kB/s | 125 kB 00:00 2024-09-11T14:34:27.698 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:27.699 INFO:teuthology.orchestra.run.smithi040.stdout:Total 683 kB/s | 125 kB 00:00 2024-09-11T14:34:27.699 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:27.704 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:27.704 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:27.772 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:27.772 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:27.932 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:28.040 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:28.749 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:28.749 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:28.749 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:28.749 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:34:28.749 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:28.749 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:29.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs2 2024-09-11T14:34:29.594 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:20 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:29.692 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:34:29.740 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:29.741 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:34:29.741 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:29.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs-devel 2024-09-11T14:34:30.409 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:21 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:30.549 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 32 k 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:30.550 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-09-11T14:34:30.551 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:30.551 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 32 k 2024-09-11T14:34:30.551 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 142 k 2024-09-11T14:34:30.551 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:30.701 INFO:teuthology.orchestra.run.smithi040.stdout:libcephfs-devel-19.3.0-4872.g0d61375b.el9.x86_6 211 kB/s | 32 kB 00:00 2024-09-11T14:34:30.701 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:30.702 INFO:teuthology.orchestra.run.smithi040.stdout:Total 209 kB/s | 32 kB 00:00 2024-09-11T14:34:30.702 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:30.708 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:30.708 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:30.737 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:30.737 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:30.826 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:30.935 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:31.291 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:31.537 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:34:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:31.537 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:31.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados2 2024-09-11T14:34:32.302 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:23 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:32.396 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:34:32.442 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:32.443 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:34:32.443 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:32.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librbd1 2024-09-11T14:34:33.097 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:24 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:33.192 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:34:33.238 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:33.240 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:34:33.240 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:33.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rados 2024-09-11T14:34:33.892 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:24 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:33.985 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:34:34.031 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:34.032 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:34:34.032 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:34.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rgw 2024-09-11T14:34:34.693 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:25 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:34.786 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:34:34.832 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:34.834 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:34:34.834 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:34.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-cephfs 2024-09-11T14:34:35.496 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:26 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:35.589 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:34:35.635 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:35.636 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:34:35.636 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:35.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rbd 2024-09-11T14:34:36.290 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:27 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:36.384 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:34:36.431 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:36.432 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-09-11T14:34:36.432 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:36.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-fuse 2024-09-11T14:34:37.091 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:28 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:37.231 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 91 k 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:37.232 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-09-11T14:34:37.233 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:37.233 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 91 k 2024-09-11T14:34:37.233 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 321 k 2024-09-11T14:34:37.233 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:37.429 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-fuse-19.3.0-4872.g0d61375b.el9.x86_64.rpm 463 kB/s | 91 kB 00:00 2024-09-11T14:34:37.429 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:37.430 INFO:teuthology.orchestra.run.smithi040.stdout:Total 460 kB/s | 91 kB 00:00 2024-09-11T14:34:37.430 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:37.440 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:37.440 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:37.507 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:37.507 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:37.660 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:37.763 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:38.140 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:38.403 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:38.404 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:38.404 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:38.404 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:34:38.404 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:38.404 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:38.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-mirror 2024-09-11T14:34:39.324 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:30 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:39.468 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:39.469 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-09-11T14:34:39.469 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:39.469 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:34:39.469 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:39.470 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:39.470 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:39.470 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-09-11T14:34:39.470 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:39.470 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 3.5 M 2024-09-11T14:34:39.470 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 30 M 2024-09-11T14:34:39.470 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:39.842 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-mirror-19.3.0-4872.g0d61375b.el9.x86_64.rpm 9.4 MB/s | 3.5 MB 00:00 2024-09-11T14:34:39.843 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:39.843 INFO:teuthology.orchestra.run.smithi040.stdout:Total 9.4 MB/s | 3.5 MB 00:00 2024-09-11T14:34:39.844 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:39.864 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:39.865 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:39.962 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:39.962 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:40.590 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:40.842 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:40.872 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:34:40.872 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T14:34:40.872 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:34:40.872 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:34:40.872 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:41.841 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:41.841 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:41.841 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:41.842 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:34:41.842 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:41.842 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:42.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-nbd 2024-09-11T14:34:42.699 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:33 ago on Wed 11 Sep 2024 02:32:09 PM UTC. 2024-09-11T14:34:42.841 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-09-11T14:34:42.841 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 177 k 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:42.842 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 177 k 2024-09-11T14:34:42.843 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 685 k 2024-09-11T14:34:42.843 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-09-11T14:34:43.062 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-nbd-19.3.0-4872.g0d61375b.el9.x86_64.rpm 807 kB/s | 177 kB 00:00 2024-09-11T14:34:43.062 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:34:43.063 INFO:teuthology.orchestra.run.smithi040.stdout:Total 802 kB/s | 177 kB 00:00 2024-09-11T14:34:43.063 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-09-11T14:34:43.074 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-09-11T14:34:43.074 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-09-11T14:34:43.143 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-09-11T14:34:43.143 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-09-11T14:34:43.302 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-09-11T14:34:43.405 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:43.838 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:44.096 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:34:44.096 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:44.096 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-09-11T14:34:44.096 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:34:44.096 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:44.096 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-09-11T14:34:44.354 DEBUG:teuthology.parallel:result is None 2024-09-11T14:34:44.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:34:44.494 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T14:34:44.516 INFO:teuthology.orchestra.run.smithi040.stdout:19.3.0-4872.g0d61375b.el9 2024-09-11T14:34:44.516 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4872.g0d61375b.el9 2024-09-11T14:34:44.517 INFO:teuthology.task.install:The correct ceph version 19.3.0-4872.g0d61375b is installed. 2024-09-11T14:34:44.518 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T14:34:44.518 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:44.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T14:34:44.582 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T14:34:44.583 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:44.583 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T14:34:44.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T14:34:44.713 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T14:34:44.714 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:44.714 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T14:34:44.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T14:34:44.846 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T14:34:44.846 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:44.846 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T14:34:44.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T14:34:44.979 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T14:34:45.068 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T14:34:45.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2024-09-11T14:34:45.100 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T14:34:45.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T14:34:45.165 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T14:34:45.166 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T14:34:45.240 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T14:34:45.240 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T14:34:45.240 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901361', 'branch': 'main', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format}', 'email': None, 'first_in_suite': False, 'job_id': '7901361', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'workunit': {'branch': 'wip-matanb-crimson-seastore-logs', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5813, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-seastore-logs', 'suite_path': '/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/matan-b/ceph.git', 'suite_sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2', 'targets': {'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEUolyCmVKyS4nDkm1L2008caDo2YChaf83PiHAnb8Hb4K4Qhiz1NQ2C8C/RcnwmoSsV6YGcLNFrmW39DHnyaNw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-11_13:56:22', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T14:34:45.241 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T14:34:45.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T14:34:45.354 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:45.355 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T14:34:45.410 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T14:34:45.411 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-09-11T14:34:45.465 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T14:34:45.466 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:34:45.466 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-11T14:34:45.466 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:34:45.466 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:34:45.466 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-09-11 14:34:41.116995225 +0000 2024-09-11T14:34:45.466 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-09-11 14:31:20.061275752 +0000 2024-09-11T14:34:45.466 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-09-11 14:31:20.061275752 +0000 2024-09-11T14:34:45.466 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-09-11 14:31:20.061275752 +0000 2024-09-11T14:34:45.466 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T14:34:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-09-11T14:34:45.530 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-09-11T14:34:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000203951 s, 2.5 MB/s 2024-09-11T14:34:45.532 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T14:34:45.588 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-09-11 14:34:41.117995211 +0000 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-09-11 14:31:20.401271140 +0000 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-09-11 14:31:20.401271140 +0000 2024-09-11T14:34:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-09-11 14:31:20.401271140 +0000 2024-09-11T14:34:45.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T14:34:45.708 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-09-11T14:34:45.709 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-09-11T14:34:45.709 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000190787 s, 2.7 MB/s 2024-09-11T14:34:45.710 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T14:34:45.766 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-09-11T14:34:45.823 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T14:34:45.823 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:34:45.823 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-11T14:34:45.823 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:34:45.824 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:34:45.824 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-09-11 14:34:41.117995211 +0000 2024-09-11T14:34:45.824 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-09-11 14:31:20.730266678 +0000 2024-09-11T14:34:45.824 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-09-11 14:31:20.730266678 +0000 2024-09-11T14:34:45.824 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-09-11 14:31:20.730266678 +0000 2024-09-11T14:34:45.824 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T14:34:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-09-11T14:34:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-09-11T14:34:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000204044 s, 2.5 MB/s 2024-09-11T14:34:45.888 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T14:34:45.944 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-09-11T14:34:46.001 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T14:34:46.001 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:34:46.001 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-11T14:34:46.001 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:34:46.001 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:34:46.001 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-09-11 14:34:41.118995197 +0000 2024-09-11T14:34:46.001 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-09-11 14:31:21.064262147 +0000 2024-09-11T14:34:46.002 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-09-11 14:31:21.064262147 +0000 2024-09-11T14:34:46.002 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-09-11 14:31:21.064262147 +0000 2024-09-11T14:34:46.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T14:34:46.065 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-09-11T14:34:46.066 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-09-11T14:34:46.066 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000207507 s, 2.5 MB/s 2024-09-11T14:34:46.067 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T14:34:46.124 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:34:46.124 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T14:34:46.124 INFO:tasks.ceph:Generating config... 2024-09-11T14:34:46.125 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-11T14:34:46.126 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-11T14:34:46.126 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-11T14:34:46.126 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-11T14:34:46.126 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-11T14:34:46.126 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T14:34:46.126 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T14:34:46.126 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T14:34:46.126 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-11T14:34:46.126 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-11T14:34:46.126 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-11T14:34:46.126 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-11T14:34:46.126 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T14:34:46.126 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-11T14:34:46.127 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-11T14:34:46.128 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-11T14:34:46.128 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-11T14:34:46.128 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:34:46.128 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-11T14:34:46.128 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-11T14:34:46.128 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-11T14:34:46.128 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-11T14:34:46.128 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-11T14:34:46.128 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-11T14:34:46.128 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T14:34:46.128 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T14:34:46.128 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-11T14:34:46.128 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-11T14:34:46.128 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-11T14:34:46.129 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T14:34:46.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T14:34:46.218 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T14:34:46.222 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T14:34:46.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:34:46.300 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.40')] 2024-09-11T14:34:46.300 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.40', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-11T14:34:46.301 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:46.301 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:34:46.356 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.40 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:34:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:34:46.448+0000 7f3f8226bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:34:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:34:46.449+0000 7f3f8226bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:34:46.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:34:46.449+0000 7f3f8226bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:34:46.451 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:34:46.451 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: generated fsid b33ea6be-da0e-4fe6-a2f2-8f9cbab56580 2024-09-11T14:34:46.451 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = quincy 2024-09-11T14:34:46.451 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 0 2024-09-11T14:34:46.451 INFO:teuthology.orchestra.run.smithi040.stdout:fsid b33ea6be-da0e-4fe6-a2f2-8f9cbab56580 2024-09-11T14:34:46.451 INFO:teuthology.orchestra.run.smithi040.stdout:last_changed 2024-09-11T14:34:46.450471+0000 2024-09-11T14:34:46.451 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-09-11T14:34:46.450471+0000 2024-09-11T14:34:46.452 INFO:teuthology.orchestra.run.smithi040.stdout:min_mon_release 17 (quincy) 2024-09-11T14:34:46.452 INFO:teuthology.orchestra.run.smithi040.stdout:election_strategy: 1 2024-09-11T14:34:46.452 INFO:teuthology.orchestra.run.smithi040.stdout:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-09-11T14:34:46.452 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-11T14:34:46.453 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:34:46.468 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b33ea6be-da0e-4fe6-a2f2-8f9cbab56580... 2024-09-11T14:34:46.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T14:34:46.552 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.553 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2024-09-11T14:34:46.554 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.555 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = 172.21.15.40 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: ms client mode = crc 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: ms cluster mode = crc 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon client mode = crc 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon cluster mode = crc 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon service mode = crc 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: ms service mode = crc 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = b33ea6be-da0e-4fe6-a2f2-8f9cbab56580 2024-09-11T14:34:46.556 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 600 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2024-09-11T14:34:46.557 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug trim objects = true 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.558 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: crimson alien thread cpu cores = 6-7 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: crimson osd obc lru size = 10 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore backref = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore cache = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore cleaner = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore device = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore epm = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore fixedkv tree = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore journal = 20 2024-09-11T14:34:46.559 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore lba = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore odata = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore ompap = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore onode = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore t = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug seastore tm = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = seastore 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: osd shutdown pgref assert = True 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-09-11T14:34:46.560 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:34:46.561 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: mon min osdmap epochs = 50 2024-09-11T14:34:46.562 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune interval = 2 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune min = 15 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune txsize = 2 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: paxos service trim min = 10 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default crimson = True 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2024-09-11T14:34:46.563 INFO:teuthology.orchestra.run.smithi040.stdout: crimson seastar cpu cores = 0-1 2024-09-11T14:34:46.564 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2024-09-11T14:34:46.564 INFO:teuthology.orchestra.run.smithi040.stdout: crimson seastar cpu cores = 2-3 2024-09-11T14:34:46.564 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2024-09-11T14:34:46.564 INFO:teuthology.orchestra.run.smithi040.stdout: crimson seastar cpu cores = 4-5 2024-09-11T14:34:46.565 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T14:34:46.565 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T14:34:46.673 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T14:34:46.673 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:46.673 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T14:34:46.689 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:46.689 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T14:34:46.745 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:34:46.745 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:46.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T14:34:46.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:34:46.821 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:46.822 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:34:46.877 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T14:34:46.877 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T14:34:46.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:34:46.978 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:34:46.981 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T14:34:46.982 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T14:34:46.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T14:34:47.032 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T14:34:47.046 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T14:34:47.046 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T14:34:47.046 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T14:34:47.071 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:34:47.072 INFO:tasks.ceph:role: osd.0 2024-09-11T14:34:47.072 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:34:47.072 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:34:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:34:47.145 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:34:47.644 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-09-11T14:34:47.646 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-09-11T14:34:47.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T14:34:47.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T14:34:47.796 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T14:34:47.863 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:34:47.863 INFO:tasks.ceph:role: osd.1 2024-09-11T14:34:47.863 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:34:47.863 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T14:34:47.938 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:34:47.938 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:34:47.938 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:34:47.938 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:34:47.938 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:34:47.938 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:34:47.938 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:34:47.938 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:34:47.939 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:34:47.939 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:34:48.396 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-09-11T14:34:48.398 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-09-11T14:34:48.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T14:34:48.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T14:34:48.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T14:34:48.591 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:34:48.591 INFO:tasks.ceph:role: osd.2 2024-09-11T14:34:48.591 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:34:48.591 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T14:34:48.665 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:34:48.665 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:34:48.665 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:34:48.665 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:34:48.665 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:34:48.666 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:34:48.666 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:34:48.666 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:34:48.666 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:34:48.666 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:34:49.149 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-09-11T14:34:49.152 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-09-11T14:34:49.152 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T14:34:49.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T14:34:49.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:34:49.669 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:34:49.670 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:49,670 seastar - Reactor backend: linux-aio 2024-09-11T14:34:49.685 INFO:teuthology.orchestra.run.smithi040.stderr:==28027==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:34:49.794 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b54f 0x10d76c 0xdd31a 0x4b7ec 0x5fe4397 0x5fe4708 0x5fe4aea 0x603fd14 0x5fbfef6 0x5fc00d0 0x5fc065c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x3968bb0 0x396a38a 0x380ac18 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:49.794 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:49.813 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:49,812 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:34:49.813 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:49,813 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-11T14:34:49.814 INFO:teuthology.orchestra.run.smithi040.stderr:DEBUG 2024-09-11 14:34:49,814 [shard 0:main] seastar - reactor::drain 2024-09-11T14:34:50.364 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:50.365 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-09-11T14:34:50.365 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-09-11T14:34:50.365 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-09-11T14:34:50.365 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:50.365 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-11T14:34:50.428 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:34:50.432 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:50,432 seastar - Reactor backend: linux-aio 2024-09-11T14:34:50.440 INFO:teuthology.orchestra.run.smithi040.stderr:==28025==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:34:50.505 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5558153 0x5fbbb77 0x5fc015a 0x5fc065c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:50.506 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:50.744 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xb9e213a 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:50.745 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:50.802 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3883525 0x3883672 0x9c2432f 0x9c24510 0x9b05d33 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:50.802 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:50.925 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4218f 0x3ef7b 0xfd531 0x95ea375 0x95eafcc 0x95eb495 0x9b01367 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:50.925 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3780733 0x3b91fc4 0x3b92268 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xba09714 0xba38a91 0xba390d8 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9ab752e 0x9b0561e 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:50.982 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:51.050 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10252a 0x10d76c 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:51.108 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:51.108 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:51.134 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x53b10 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:51.134 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:51.215 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4aaff 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:51.216 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:51.764 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab3d 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:51.764 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:52.822 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:52.822 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-09-11T14:34:52.822 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-09-11T14:34:52.822 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-09-11T14:34:52.822 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:52.823 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-11T14:34:52.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T14:34:52.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:34:53.186 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:34:53.187 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:53,187 seastar - Reactor backend: linux-aio 2024-09-11T14:34:53.193 INFO:teuthology.orchestra.run.smithi040.stderr:==28073==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:34:53.302 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4b89e 0x5fe4397 0x5fe49cf 0x5fe4af5 0x603fd14 0x5fbfef6 0x5fc042a 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x3968bb0 0x396a38a 0x380ac18 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:53.302 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:53.321 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:53,320 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:34:53.321 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:53,320 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-11T14:34:53.322 INFO:teuthology.orchestra.run.smithi040.stderr:DEBUG 2024-09-11 14:34:53,321 [shard 0:main] seastar - reactor::drain 2024-09-11T14:34:53.852 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:53.852 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-09-11T14:34:53.852 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-09-11T14:34:53.853 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-09-11T14:34:53.853 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:53.853 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-11T14:34:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:34:53.918 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:53,918 seastar - Reactor backend: linux-aio 2024-09-11T14:34:53.926 INFO:teuthology.orchestra.run.smithi040.stderr:==28071==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:34:53.992 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4abc2 0x5ff0dd6 0x5ff0fb8 0x5ff11cf 0x5ff14a5 0x603fba3 0x603fda9 0x5fbfef6 0x5fc00d0 0x5fc065c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:53.992 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.232 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xba3b993 0xb9d61d4 0x984ae51 0x9b0161a 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:54.232 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10251f 0x10d76c 0xdd31a 0x4ab1c 0x3619f03 0x3630204 0x38311c8 0x454eaf8 0xb9d982a 0xb9d9bd4 0xb9dfb97 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b05f61 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.316 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xba40a43 0xba40b6e 0xb9e0f68 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:54.316 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.374 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133d18 0xdd332 0x4b14c 0x3b920e7 0x3b92268 0xb9f768b 0xb9f7940 0xb9dfd11 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b024e4 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:54.374 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.412 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5ccaaaa 0x5ccabe2 0x963e4bc 0x9b01760 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:54.412 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xb9f5d7d 0xb9d647a 0x984ae51 0x9b05eb6 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.526 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554929d 0x9e3b019 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:54.526 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x390ff 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:54.584 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:54.617 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10251f 0x10d76c 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:54.617 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:56.308 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:56.308 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-09-11T14:34:56.308 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-09-11T14:34:56.308 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-09-11T14:34:56.308 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:56.308 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-11T14:34:56.334 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T14:34:56.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:34:56.658 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:34:56.659 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:56,659 seastar - Reactor backend: linux-aio 2024-09-11T14:34:56.665 INFO:teuthology.orchestra.run.smithi040.stderr:==28118==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:34:56.774 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b558 0x10d76c 0xdd31a 0x4ab1c 0x5ff0dd6 0x5ff0f9e 0x5ff11cf 0x5ff14a5 0x603fba3 0x603fda9 0x5fbfef6 0x5fc042a 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x3968bb0 0x396a38a 0x380ac18 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:56.774 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:56,794 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:34:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:56,794 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-11T14:34:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:DEBUG 2024-09-11 14:34:56,795 [shard 0:main] seastar - reactor::drain 2024-09-11T14:34:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-09-11T14:34:57.340 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-09-11T14:34:57.340 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-09-11T14:34:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:57.341 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-11T14:34:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:34:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-09-11 14:34:57,403 seastar - Reactor backend: linux-aio 2024-09-11T14:34:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:==28116==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:34:57.481 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3632828 0x3632a8c 0x5fb7f7b 0x5fd997c 0x5fd9c16 0x5fc083c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:57.481 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10a2ca 0xfd59e 0xb85ab4d 0xb85ac00 0xba2e325 0xba2e414 0xba2e694 0xba2e74d 0xba2ec3f 0xba2f101 0xb9d3249 0xba2fa98 0xba2fcdf 0xba2fd8f 0xb9df75d 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4bf8d 0x36e8231 0xba29217 0xba29498 0xba2964e 0xba2979b 0xb9d1e3e 0xb9d64f1 0x984ae51 0x9b05eb6 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xb9f3c94 0xb9f4222 0xb9e014c 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x53b10 0x4b14c 0xba3c24e 0xba3c388 0xba3c71e 0xba3c930 0xb9d97af 0xb9d9bd4 0xb9dfb97 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9ab9cb7 0x9b05946 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:57.897 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3780709 0x3acda78 0x3b9225d 0xba052ce 0xba407e3 0xba40b6e 0xb9e0f68 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:57.897 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x357e49f 0x3acdafe 0x3b9225d 0xb9f768b 0xb9fb0f9 0xba3eb1c 0xb9dfda2 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9ab9cb7 0x9b05946 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:58.007 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x156dc 0xdd312 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:58.007 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:58.065 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549429 0x9e3b019 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:34:58.065 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x9e3b026 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:34:58.105 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-09-11T14:34:59.789 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:59.790 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-09-11T14:34:59.790 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-09-11T14:34:59.790 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-09-11T14:34:59.790 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-09-11T14:34:59.790 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-11T14:34:59.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T14:34:59.841 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T14:34:59.842 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:59.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T14:34:59.907 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:59.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T14:34:59.972 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:34:59.972 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T14:35:00.037 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:35:00.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T14:35:00.102 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:35:00.103 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T14:35:00.159 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T14:35:00.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T14:35:00.223 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2024-09-11T14:35:00.223 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCGquFmnn9POhAABaVFOJnWgyA2k3x3a8jHtg== 2024-09-11T14:35:00.223 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2024-09-11T14:35:00.223 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCKquFmIgpbIxAAVnk1tC68xSx0lJ+D14AS6w== 2024-09-11T14:35:00.223 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2024-09-11T14:35:00.223 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCOquFmcRPSBBAAZOsfw+6Xn5nXiQifsYgxDg== 2024-09-11T14:35:00.224 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2024-09-11T14:35:00.224 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCRquFmJl3xIRAAoNUkXpvCbq4O+9TpAKgX1w== 2024-09-11T14:35:00.224 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-09-11T14:35:00.224 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQCHquFm9ePsARAA6GTuWKNXFRWJEYNDRyyIcg== 2024-09-11T14:35:00.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T14:35:00.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:35:00.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:35:00.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:35:00.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T14:35:00.582 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T14:35:00.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T14:35:00.608 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T14:35:00.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:00.715+0000 7f8d84eebc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:00.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:00.715+0000 7f8d84eebc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:00.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T14:35:00.756 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:35:00.816 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T14:35:00.816 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T14:35:00.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T14:35:00.858 INFO:tasks.ceph.mon.a:Started 2024-09-11T14:35:00.859 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T14:35:00.859 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T14:35:00.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T14:35:00.861 INFO:tasks.ceph.mgr.x:Started 2024-09-11T14:35:00.861 DEBUG:tasks.ceph:set 0 configs 2024-09-11T14:35:00.861 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T14:35:00.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T14:35:00.945 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-09-11T14:35:00.943+0000 7f635622dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:00.945 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-09-11T14:35:00.943+0000 7f635622dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:00.945 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-09-11T14:35:00.943+0000 7f635622dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:00.947 INFO:tasks.ceph.mgr.x.smithi040.stderr:2024-09-11T14:35:00.946+0000 7fbaede33000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:00.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:00.969+0000 7f0ac6d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:00.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:00.980+0000 7f0ac6d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-09-11T14:35:01.067 INFO:tasks.ceph:check_enable_crimson: True 2024-09-11T14:35:01.067 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-11T14:35:01.067 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-11T14:35:01.154 INFO:tasks.ceph.mgr.x.smithi040.stderr:2024-09-11T14:35:01.152+0000 7fbaede33000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:01.154 INFO:tasks.ceph.mgr.x.smithi040.stderr:2024-09-11T14:35:01.152+0000 7fbaede33000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:01.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:01.171+0000 7fd2f0ea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:01.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:01.180+0000 7fd2f0ea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:01.265 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-09-11T14:35:01.264+0000 7f634cbf4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-11T14:35:01.265 INFO:tasks.ceph.mon.a.smithi040.stderr:Please be aware that this feature is experimental, untested, 2024-09-11T14:35:01.266 INFO:tasks.ceph.mon.a.smithi040.stderr:unsupported, and may result in data corruption, data loss, 2024-09-11T14:35:01.266 INFO:tasks.ceph.mon.a.smithi040.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-11T14:35:01.266 INFO:tasks.ceph.mon.a.smithi040.stderr:feature with important data. 2024-09-11T14:35:01.266 INFO:tasks.ceph.mon.a.smithi040.stderr: 2024-09-11T14:35:02.068 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T14:35:02.068 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:35:02.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T14:35:02.094 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:35:02.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T14:35:02.160 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:35:02.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T14:35:02.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 9fb327d3-b49a-4c74-be15-20ef7244a835 0 2024-09-11T14:35:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:02.370+0000 7f6a5d4d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:02.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:02.381+0000 7f6a5d4d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:02.464 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-09-11T14:35:02.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 0ba9f622-b79e-472e-8c04-96b768f7bc6e 1 2024-09-11T14:35:02.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:02.580+0000 7f5d57e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:02.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:02.591+0000 7f5d57e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:02.673 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-09-11T14:35:02.684 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new abd80b24-fbd4-4e2e-bdb6-8b8052d07ac4 2 2024-09-11T14:35:02.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:02.788+0000 7fadeb45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:02.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:02.799+0000 7fadeb45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:02.880 INFO:teuthology.orchestra.run.smithi040.stdout:2 2024-09-11T14:35:02.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-11T14:35:02.923 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T14:35:02.923 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-11T14:35:02.965 INFO:tasks.ceph.osd.0:Started 2024-09-11T14:35:02.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-11T14:35:02.997 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T14:35:02.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-11T14:35:03.040 INFO:tasks.ceph.osd.1:Started 2024-09-11T14:35:03.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-11T14:35:03.074 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T14:35:03.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-11T14:35:03.117 INFO:tasks.ceph.osd.2:Started 2024-09-11T14:35:03.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:35:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:03.281+0000 7f9855791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:03.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:03.294+0000 7f9855791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:03.371 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:35:03.371 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":6,"fsid":"b33ea6be-da0e-4fe6-a2f2-8f9cbab56580","created":"2024-09-11T14:35:00.967980+0000","modified":"2024-09-11T14:35:02.879246+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:35:02.879246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9fb327d3-b49a-4c74-be15-20ef7244a835","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0ba9f622-b79e-472e-8c04-96b768f7bc6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"abd80b24-fbd4-4e2e-bdb6-8b8052d07ac4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:35:03.383 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T14:35:03.383 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T14:35:03.685 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:35:03.794 INFO:teuthology.misc.health.smithi040.stderr:2024-09-11T14:35:03.788+0000 7f4c0caec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:03.799 INFO:teuthology.misc.health.smithi040.stderr:2024-09-11T14:35:03.798+0000 7f4c0caec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:03.873 INFO:teuthology.misc.health.smithi040.stdout: 2024-09-11T14:35:03.873 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":6,"fsid":"b33ea6be-da0e-4fe6-a2f2-8f9cbab56580","created":"2024-09-11T14:35:00.967980+0000","modified":"2024-09-11T14:35:02.879246+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:35:02.879246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9fb327d3-b49a-4c74-be15-20ef7244a835","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0ba9f622-b79e-472e-8c04-96b768f7bc6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"abd80b24-fbd4-4e2e-bdb6-8b8052d07ac4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:35:03.884 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T14:35:06.748 INFO:tasks.ceph.mgr.x.smithi040.stderr:2024-09-11T14:35:06.746+0000 7fba8ad04640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T14:35:10.192 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:35:10.301 INFO:teuthology.misc.health.smithi040.stderr:2024-09-11T14:35:10.295+0000 7fca104e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:10.306 INFO:teuthology.misc.health.smithi040.stderr:2024-09-11T14:35:10.304+0000 7fca104e3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:10.507 INFO:teuthology.misc.health.smithi040.stdout: 2024-09-11T14:35:10.508 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"b33ea6be-da0e-4fe6-a2f2-8f9cbab56580","created":"2024-09-11T14:35:00.967980+0000","modified":"2024-09-11T14:35:09.830264+0000","last_up_change":"2024-09-11T14:35:08.819126+0000","last_in_change":"2024-09-11T14:35:02.879246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:35:09.770562+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9fb327d3-b49a-4c74-be15-20ef7244a835","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1512393070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":1512393070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1512393070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1512393070}]},"public_addr":"172.21.15.40:6808/1512393070","cluster_addr":"172.21.15.40:6809/1512393070","heartbeat_back_addr":"172.21.15.40:6806/1512393070","heartbeat_front_addr":"172.21.15.40:6818/1512393070","state":["exists","up"]},{"osd":1,"uuid":"0ba9f622-b79e-472e-8c04-96b768f7bc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":991486483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":991486483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":991486483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":991486483}]},"public_addr":"172.21.15.40:6803/991486483","cluster_addr":"172.21.15.40:6804/991486483","heartbeat_back_addr":"172.21.15.40:6802/991486483","heartbeat_front_addr":"172.21.15.40:6805/991486483","state":["exists","up"]},{"osd":2,"uuid":"abd80b24-fbd4-4e2e-bdb6-8b8052d07ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3190718975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":3190718975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3190718975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":3190718975}]},"public_addr":"172.21.15.40:6810/3190718975","cluster_addr":"172.21.15.40:6807/3190718975","heartbeat_back_addr":"172.21.15.40:6816/3190718975","heartbeat_front_addr":"172.21.15.40:6817/3190718975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:35:10.521 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T14:35:10.521 INFO:tasks.ceph:Creating RBD pool 2024-09-11T14:35:10.521 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T14:35:10.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:10.630+0000 7f2705991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:10.642+0000 7f2705991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'rbd' created 2024-09-11T14:35:11.865 DEBUG:teuthology.orchestra.run.smithi040:> rbd --cluster ceph pool init rbd 2024-09-11T14:35:11.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:11.960+0000 7f524a00eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:11.960+0000 7f524a00eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:11.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:11.967+0000 7f524a00eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:14.979 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T14:35:14.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T14:35:14.980 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T14:35:15.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:15.089+0000 7f08b9c74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:15.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:15.098+0000 7f08b9c74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:15.294 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:35:15.307 INFO:teuthology.orchestra.run.smithi040.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-11T14:35:15.308 INFO:tasks.ceph_manager:config epoch is 1 2024-09-11T14:35:15.308 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T14:35:15.308 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T14:35:15.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T14:35:15.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:15.415+0000 7f3164148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:15.424+0000 7f3164148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:15.675 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:35:15.687 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3901873376},{"type":"v1","addr":"172.21.15.40:6801","nonce":3901873376}]},"active_addr":"172.21.15.40:6801/3901873376","active_change":"2024-09-11T14:35:05.731446+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":1239806888}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2278162229}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":4122739171}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":1498074946}]}]} 2024-09-11T14:35:15.689 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T14:35:15.689 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T14:35:15.690 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:35:15.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:15.797+0000 7fedc6551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:15.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:15.805+0000 7fedc6551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.007 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:35:16.007 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":15,"fsid":"b33ea6be-da0e-4fe6-a2f2-8f9cbab56580","created":"2024-09-11T14:35:00.967980+0000","modified":"2024-09-11T14:35:14.897886+0000","last_up_change":"2024-09-11T14:35:08.819126+0000","last_in_change":"2024-09-11T14:35:02.879246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:35:09.770562+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:35:10.863372+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9fb327d3-b49a-4c74-be15-20ef7244a835","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1512393070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":1512393070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1512393070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1512393070}]},"public_addr":"172.21.15.40:6808/1512393070","cluster_addr":"172.21.15.40:6809/1512393070","heartbeat_back_addr":"172.21.15.40:6806/1512393070","heartbeat_front_addr":"172.21.15.40:6818/1512393070","state":["exists","up"]},{"osd":1,"uuid":"0ba9f622-b79e-472e-8c04-96b768f7bc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":991486483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":991486483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":991486483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":991486483}]},"public_addr":"172.21.15.40:6803/991486483","cluster_addr":"172.21.15.40:6804/991486483","heartbeat_back_addr":"172.21.15.40:6802/991486483","heartbeat_front_addr":"172.21.15.40:6805/991486483","state":["exists","up"]},{"osd":2,"uuid":"abd80b24-fbd4-4e2e-bdb6-8b8052d07ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3190718975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":3190718975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3190718975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":3190718975}]},"public_addr":"172.21.15.40:6810/3190718975","cluster_addr":"172.21.15.40:6807/3190718975","heartbeat_back_addr":"172.21.15.40:6816/3190718975","heartbeat_front_addr":"172.21.15.40:6817/3190718975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:35:16.020 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T14:35:16.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:35:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.126+0000 7f343f311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.134+0000 7f343f311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.333 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:35:16.333 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":15,"fsid":"b33ea6be-da0e-4fe6-a2f2-8f9cbab56580","created":"2024-09-11T14:35:00.967980+0000","modified":"2024-09-11T14:35:14.897886+0000","last_up_change":"2024-09-11T14:35:08.819126+0000","last_in_change":"2024-09-11T14:35:02.879246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:35:09.770562+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:35:10.863372+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9fb327d3-b49a-4c74-be15-20ef7244a835","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1512393070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":1512393070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1512393070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1512393070}]},"public_addr":"172.21.15.40:6808/1512393070","cluster_addr":"172.21.15.40:6809/1512393070","heartbeat_back_addr":"172.21.15.40:6806/1512393070","heartbeat_front_addr":"172.21.15.40:6818/1512393070","state":["exists","up"]},{"osd":1,"uuid":"0ba9f622-b79e-472e-8c04-96b768f7bc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":991486483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":991486483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":991486483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":991486483}]},"public_addr":"172.21.15.40:6803/991486483","cluster_addr":"172.21.15.40:6804/991486483","heartbeat_back_addr":"172.21.15.40:6802/991486483","heartbeat_front_addr":"172.21.15.40:6805/991486483","state":["exists","up"]},{"osd":2,"uuid":"abd80b24-fbd4-4e2e-bdb6-8b8052d07ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3190718975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":3190718975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3190718975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":3190718975}]},"public_addr":"172.21.15.40:6810/3190718975","cluster_addr":"172.21.15.40:6807/3190718975","heartbeat_back_addr":"172.21.15.40:6816/3190718975","heartbeat_front_addr":"172.21.15.40:6817/3190718975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:35:16.350 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T14:35:16.350 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T14:35:16.351 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T14:35:16.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.464+0000 7fc596288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.464+0000 7fe19a49c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.464+0000 7f93caa9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.474+0000 7f93caa9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.474+0000 7fe19a49c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.474+0000 7fc596288640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.505 INFO:teuthology.orchestra.run.smithi040.stdout:34359738376 2024-09-11T14:35:16.506 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:35:16.511 INFO:teuthology.orchestra.run.smithi040.stdout:38654705671 2024-09-11T14:35:16.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:35:16.512 INFO:teuthology.orchestra.run.smithi040.stdout:38654705671 2024-09-11T14:35:16.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:35:16.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.614+0000 7f3cd5791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.625+0000 7f3cd5791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.664+0000 7f549d4c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.674+0000 7f549d4c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.676+0000 7f1418791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:16.687+0000 7f1418791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:16.849 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2024-09-11T14:35:16.863 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738373 for osd.1 2024-09-11T14:35:16.877 INFO:teuthology.orchestra.run.smithi040.stdout:38654705668 2024-09-11T14:35:16.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.0 2024-09-11T14:35:16.896 INFO:teuthology.orchestra.run.smithi040.stdout:38654705668 2024-09-11T14:35:16.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.2 2024-09-11T14:35:17.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:35:17.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:35:17.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:35:17.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:17.990+0000 7f6c45551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.002+0000 7f6c45551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.027+0000 7f5d75711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.039+0000 7f5d75711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.067+0000 7fc286f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.076+0000 7fc286f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.219 INFO:teuthology.orchestra.run.smithi040.stdout:34359738376 2024-09-11T14:35:18.237 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738376 for osd.1 2024-09-11T14:35:18.237 DEBUG:teuthology.parallel:result is None 2024-09-11T14:35:18.259 INFO:teuthology.orchestra.run.smithi040.stdout:38654705671 2024-09-11T14:35:18.273 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.0 2024-09-11T14:35:18.273 DEBUG:teuthology.parallel:result is None 2024-09-11T14:35:18.290 INFO:teuthology.orchestra.run.smithi040.stdout:38654705671 2024-09-11T14:35:18.303 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.2 2024-09-11T14:35:18.304 DEBUG:teuthology.parallel:result is None 2024-09-11T14:35:18.304 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:35:18.304 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:35:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.451+0000 7f927c191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.459+0000 7f927c191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:35:18.657 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-09-11T14:35:18.670 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-11T14:35:17.740833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3372,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889812,"statfs":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.844118"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.927279+0000","last_change":"2024-09-11T14:35:14.942992+0000","last_active":"2024-09-11T14:35:14.927279+0000","last_peered":"2024-09-11T14:35:14.927279+0000","last_clean":"2024-09-11T14:35:14.927279+0000","last_became_active":"2024-09-11T14:35:12.958460+0000","last_became_peered":"2024-09-11T14:35:12.958460+0000","last_unstale":"2024-09-11T14:35:14.927279+0000","last_undegraded":"2024-09-11T14:35:14.927279+0000","last_fullsized":"2024-09-11T14:35:14.927279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.938676+0000","last_change":"2024-09-11T14:35:14.946690+0000","last_active":"2024-09-11T14:35:14.938676+0000","last_peered":"2024-09-11T14:35:14.938676+0000","last_clean":"2024-09-11T14:35:14.938676+0000","last_became_active":"2024-09-11T14:35:13.015169+0000","last_became_peered":"2024-09-11T14:35:13.015169+0000","last_unstale":"2024-09-11T14:35:14.938676+0000","last_undegraded":"2024-09-11T14:35:14.938676+0000","last_fullsized":"2024-09-11T14:35:14.938676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.931756+0000","last_change":"2024-09-11T14:35:14.942753+0000","last_active":"2024-09-11T14:35:14.931756+0000","last_peered":"2024-09-11T14:35:14.931756+0000","last_clean":"2024-09-11T14:35:14.931756+0000","last_became_active":"2024-09-11T14:35:12.989419+0000","last_became_peered":"2024-09-11T14:35:12.989419+0000","last_unstale":"2024-09-11T14:35:14.931756+0000","last_undegraded":"2024-09-11T14:35:14.931756+0000","last_fullsized":"2024-09-11T14:35:14.931756+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.939813+0000","last_change":"2024-09-11T14:35:14.947434+0000","last_active":"2024-09-11T14:35:14.939813+0000","last_peered":"2024-09-11T14:35:14.939813+0000","last_clean":"2024-09-11T14:35:14.939813+0000","last_became_active":"2024-09-11T14:35:12.976308+0000","last_became_peered":"2024-09-11T14:35:12.976308+0000","last_unstale":"2024-09-11T14:35:14.939813+0000","last_undegraded":"2024-09-11T14:35:14.939813+0000","last_fullsized":"2024-09-11T14:35:14.939813+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.951771+0000","last_change":"2024-09-11T14:35:14.951771+0000","last_active":"2024-09-11T14:35:14.951771+0000","last_peered":"2024-09-11T14:35:14.951771+0000","last_clean":"2024-09-11T14:35:14.951771+0000","last_became_active":"2024-09-11T14:35:12.981976+0000","last_became_peered":"2024-09-11T14:35:12.981976+0000","last_unstale":"2024-09-11T14:35:14.951771+0000","last_undegraded":"2024-09-11T14:35:14.951771+0000","last_fullsized":"2024-09-11T14:35:14.951771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.946145+0000","last_change":"2024-09-11T14:35:14.951268+0000","last_active":"2024-09-11T14:35:14.946145+0000","last_peered":"2024-09-11T14:35:14.946145+0000","last_clean":"2024-09-11T14:35:14.946145+0000","last_became_active":"2024-09-11T14:35:12.945557+0000","last_became_peered":"2024-09-11T14:35:12.945557+0000","last_unstale":"2024-09-11T14:35:14.946145+0000","last_undegraded":"2024-09-11T14:35:14.946145+0000","last_fullsized":"2024-09-11T14:35:14.946145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.938105+0000","last_change":"2024-09-11T14:35:14.944204+0000","last_active":"2024-09-11T14:35:14.938105+0000","last_peered":"2024-09-11T14:35:14.938105+0000","last_clean":"2024-09-11T14:35:14.938105+0000","last_became_active":"2024-09-11T14:35:12.965045+0000","last_became_peered":"2024-09-11T14:35:12.965045+0000","last_unstale":"2024-09-11T14:35:14.938105+0000","last_undegraded":"2024-09-11T14:35:14.938105+0000","last_fullsized":"2024-09-11T14:35:14.938105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.928771+0000","last_change":"2024-09-11T14:35:14.942049+0000","last_active":"2024-09-11T14:35:14.928771+0000","last_peered":"2024-09-11T14:35:14.928771+0000","last_clean":"2024-09-11T14:35:14.928771+0000","last_became_active":"2024-09-11T14:35:12.944374+0000","last_became_peered":"2024-09-11T14:35:12.944374+0000","last_unstale":"2024-09-11T14:35:14.928771+0000","last_undegraded":"2024-09-11T14:35:14.928771+0000","last_fullsized":"2024-09-11T14:35:14.928771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.934735+0000","last_change":"2024-09-11T14:35:10.963766+0000","last_active":"2024-09-11T14:35:14.934735+0000","last_peered":"2024-09-11T14:35:14.934735+0000","last_clean":"2024-09-11T14:35:14.934735+0000","last_became_active":"2024-09-11T14:35:10.942660+0000","last_became_peered":"2024-09-11T14:35:10.942660+0000","last_unstale":"2024-09-11T14:35:14.934735+0000","last_undegraded":"2024-09-11T14:35:14.934735+0000","last_fullsized":"2024-09-11T14:35:14.934735+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_clean_scrub_stamp":"2024-09-11T14:35:09.830264+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297292,"statfs":{"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1596,"kb_used_data":1596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296132,"statfs":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1340,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296388,"statfs":{"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:35:18.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:35:18.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.778+0000 7f6428074640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:18.786+0000 7f6428074640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:35:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-09-11T14:35:18.994 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-11T14:35:17.740833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3372,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889812,"statfs":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.844118"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.927279+0000","last_change":"2024-09-11T14:35:14.942992+0000","last_active":"2024-09-11T14:35:14.927279+0000","last_peered":"2024-09-11T14:35:14.927279+0000","last_clean":"2024-09-11T14:35:14.927279+0000","last_became_active":"2024-09-11T14:35:12.958460+0000","last_became_peered":"2024-09-11T14:35:12.958460+0000","last_unstale":"2024-09-11T14:35:14.927279+0000","last_undegraded":"2024-09-11T14:35:14.927279+0000","last_fullsized":"2024-09-11T14:35:14.927279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.938676+0000","last_change":"2024-09-11T14:35:14.946690+0000","last_active":"2024-09-11T14:35:14.938676+0000","last_peered":"2024-09-11T14:35:14.938676+0000","last_clean":"2024-09-11T14:35:14.938676+0000","last_became_active":"2024-09-11T14:35:13.015169+0000","last_became_peered":"2024-09-11T14:35:13.015169+0000","last_unstale":"2024-09-11T14:35:14.938676+0000","last_undegraded":"2024-09-11T14:35:14.938676+0000","last_fullsized":"2024-09-11T14:35:14.938676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.931756+0000","last_change":"2024-09-11T14:35:14.942753+0000","last_active":"2024-09-11T14:35:14.931756+0000","last_peered":"2024-09-11T14:35:14.931756+0000","last_clean":"2024-09-11T14:35:14.931756+0000","last_became_active":"2024-09-11T14:35:12.989419+0000","last_became_peered":"2024-09-11T14:35:12.989419+0000","last_unstale":"2024-09-11T14:35:14.931756+0000","last_undegraded":"2024-09-11T14:35:14.931756+0000","last_fullsized":"2024-09-11T14:35:14.931756+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.939813+0000","last_change":"2024-09-11T14:35:14.947434+0000","last_active":"2024-09-11T14:35:14.939813+0000","last_peered":"2024-09-11T14:35:14.939813+0000","last_clean":"2024-09-11T14:35:14.939813+0000","last_became_active":"2024-09-11T14:35:12.976308+0000","last_became_peered":"2024-09-11T14:35:12.976308+0000","last_unstale":"2024-09-11T14:35:14.939813+0000","last_undegraded":"2024-09-11T14:35:14.939813+0000","last_fullsized":"2024-09-11T14:35:14.939813+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.951771+0000","last_change":"2024-09-11T14:35:14.951771+0000","last_active":"2024-09-11T14:35:14.951771+0000","last_peered":"2024-09-11T14:35:14.951771+0000","last_clean":"2024-09-11T14:35:14.951771+0000","last_became_active":"2024-09-11T14:35:12.981976+0000","last_became_peered":"2024-09-11T14:35:12.981976+0000","last_unstale":"2024-09-11T14:35:14.951771+0000","last_undegraded":"2024-09-11T14:35:14.951771+0000","last_fullsized":"2024-09-11T14:35:14.951771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.946145+0000","last_change":"2024-09-11T14:35:14.951268+0000","last_active":"2024-09-11T14:35:14.946145+0000","last_peered":"2024-09-11T14:35:14.946145+0000","last_clean":"2024-09-11T14:35:14.946145+0000","last_became_active":"2024-09-11T14:35:12.945557+0000","last_became_peered":"2024-09-11T14:35:12.945557+0000","last_unstale":"2024-09-11T14:35:14.946145+0000","last_undegraded":"2024-09-11T14:35:14.946145+0000","last_fullsized":"2024-09-11T14:35:14.946145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.938105+0000","last_change":"2024-09-11T14:35:14.944204+0000","last_active":"2024-09-11T14:35:14.938105+0000","last_peered":"2024-09-11T14:35:14.938105+0000","last_clean":"2024-09-11T14:35:14.938105+0000","last_became_active":"2024-09-11T14:35:12.965045+0000","last_became_peered":"2024-09-11T14:35:12.965045+0000","last_unstale":"2024-09-11T14:35:14.938105+0000","last_undegraded":"2024-09-11T14:35:14.938105+0000","last_fullsized":"2024-09-11T14:35:14.938105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.928771+0000","last_change":"2024-09-11T14:35:14.942049+0000","last_active":"2024-09-11T14:35:14.928771+0000","last_peered":"2024-09-11T14:35:14.928771+0000","last_clean":"2024-09-11T14:35:14.928771+0000","last_became_active":"2024-09-11T14:35:12.944374+0000","last_became_peered":"2024-09-11T14:35:12.944374+0000","last_unstale":"2024-09-11T14:35:14.928771+0000","last_undegraded":"2024-09-11T14:35:14.928771+0000","last_fullsized":"2024-09-11T14:35:14.928771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:35:14.934735+0000","last_change":"2024-09-11T14:35:10.963766+0000","last_active":"2024-09-11T14:35:14.934735+0000","last_peered":"2024-09-11T14:35:14.934735+0000","last_clean":"2024-09-11T14:35:14.934735+0000","last_became_active":"2024-09-11T14:35:10.942660+0000","last_became_peered":"2024-09-11T14:35:10.942660+0000","last_unstale":"2024-09-11T14:35:14.934735+0000","last_undegraded":"2024-09-11T14:35:14.934735+0000","last_fullsized":"2024-09-11T14:35:14.934735+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_clean_scrub_stamp":"2024-09-11T14:35:09.830264+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297292,"statfs":{"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1596,"kb_used_data":1596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296132,"statfs":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1340,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296388,"statfs":{"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:35:18.995 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:35:18.995 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T14:35:18.995 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T14:35:18.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T14:35:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:19.102+0000 7f431049c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:35:19.109+0000 7f431049c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:35:19.343 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:35:19.343 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T14:35:19.356 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T14:35:19.356 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T14:35:19.372 INFO:tasks.workunit:Pulling workunits from ref dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 2024-09-11T14:35:19.372 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T14:35:19.373 DEBUG:teuthology.orchestra.run.smithi040:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:35:19.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:35:19.388 INFO:teuthology.orchestra.run.smithi040.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T14:35:19.389 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:35:19.445 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-11T14:35:19.445 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-11T14:35:19.501 INFO:tasks.workunit:timeout=3h 2024-09-11T14:35:19.501 INFO:tasks.workunit:cleanup=True 2024-09-11T14:35:19.502 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/matan-b/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 2024-09-11T14:35:19.558 INFO:tasks.workunit.client.0.smithi040.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T14:36:04.619 INFO:tasks.workunit.client.0.smithi040.stderr:Note: switching to 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'. 2024-09-11T14:36:04.619 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-09-11T14:36:04.619 INFO:tasks.workunit.client.0.smithi040.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T14:36:04.619 INFO:tasks.workunit.client.0.smithi040.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr: git switch -c 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr:Or undo this operation with: 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr: git switch - 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-09-11T14:36:04.620 INFO:tasks.workunit.client.0.smithi040.stderr:HEAD is now at dbebf1a3db6 qa/config/seastore: enable all subsystem debug 2024-09-11T14:36:04.623 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T14:36:04.685 INFO:tasks.workunit.client.0.smithi040.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T14:36:04.688 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T14:36:04.688 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T14:36:05.313 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T14:36:05.387 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T14:36:05.427 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T14:36:05.429 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T14:36:05.429 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T14:36:05.469 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T14:36:05.472 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-11T14:36:05.472 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T14:36:05.528 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2024-09-11T14:36:05.530 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2024-09-11T14:36:05.530 DEBUG:teuthology.orchestra.run.smithi040:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-09-11T14:36:05.640 INFO:tasks.workunit.client.0.smithi040.stdout:seed 30119 2024-09-11T14:36:05.640 INFO:tasks.workunit.client.0.smithi040.stdout:Note: Google Test filter = TestLibRBD.* 2024-09-11T14:36:05.640 INFO:tasks.workunit.client.0.smithi040.stdout:[==========] Running 171 tests from 1 test suite. 2024-09-11T14:36:05.640 INFO:tasks.workunit.client.0.smithi040.stdout:[----------] Global test environment set-up. 2024-09-11T14:36:05.640 INFO:tasks.workunit.client.0.smithi040.stdout:[----------] 171 tests from TestLibRBD 2024-09-11T14:36:05.690 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-09-11T14:36:06.988 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:09.185 INFO:tasks.workunit.client.0.smithi040.stdout:image has size 2097152 and order 22 2024-09-11T14:36:09.185 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.CreateAndStat (3494 ms) 2024-09-11T14:36:09.185 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-09-11T14:36:09.185 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:780: Skipped 2024-09-11T14:36:09.185 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:36:09.185 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool (0 ms) 2024-09-11T14:36:09.185 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-09-11T14:36:09.186 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:09.320 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.CreateAndStatPP (135 ms) 2024-09-11T14:36:09.320 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.GetId 2024-09-11T14:36:09.320 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:09.434 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.GetId (115 ms) 2024-09-11T14:36:09.435 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-09-11T14:36:09.435 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:09.566 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.GetIdPP (131 ms) 2024-09-11T14:36:09.566 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-09-11T14:36:09.566 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:09.681 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (115 ms) 2024-09-11T14:36:09.681 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-09-11T14:36:09.681 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (123 ms) 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:939: Skipped 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp (0 ms) 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:962: Skipped 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP (0 ms) 2024-09-11T14:36:09.804 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.OpenAio 2024-09-11T14:36:09.805 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:09.921 INFO:tasks.workunit.client.0.smithi040.stdout:image has size 2097152 and order 22 2024-09-11T14:36:09.922 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.OpenAio (118 ms) 2024-09-11T14:36:09.922 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-09-11T14:36:09.935 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.OpenAioFail (13 ms) 2024-09-11T14:36:09.935 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-09-11T14:36:09.935 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:10.124 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.OpenAioPP (189 ms) 2024-09-11T14:36:10.124 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-09-11T14:36:10.138 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.OpenAioFailPP (15 ms) 2024-09-11T14:36:10.138 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-09-11T14:36:10.138 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:10.421 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.ResizeAndStat (282 ms) 2024-09-11T14:36:10.421 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-09-11T14:36:10.421 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:10.646 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (226 ms) 2024-09-11T14:36:10.646 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-09-11T14:36:10.647 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:10.917 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (270 ms) 2024-09-11T14:36:10.917 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-09-11T14:36:10.917 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:11.199 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (283 ms) 2024-09-11T14:36:11.199 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-09-11T14:36:13.322 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:15.192 INFO:tasks.workunit.client.0.smithi040.stdout:image: image15 2024-09-11T14:36:15.192 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image15 2024-09-11T14:36:15.192 INFO:tasks.workunit.client.0.smithi040.stdout:found image15 2024-09-11T14:36:15.193 INFO:tasks.workunit.client.0.smithi040.stdout:erased image15 2024-09-11T14:36:15.193 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:15.536 INFO:tasks.workunit.client.0.smithi040.stdout:image: image15 2024-09-11T14:36:15.536 INFO:tasks.workunit.client.0.smithi040.stdout:image: image16 2024-09-11T14:36:15.536 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image15 2024-09-11T14:36:15.536 INFO:tasks.workunit.client.0.smithi040.stdout:found image15 2024-09-11T14:36:15.536 INFO:tasks.workunit.client.0.smithi040.stdout:erased image15 2024-09-11T14:36:15.536 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image16 2024-09-11T14:36:15.536 INFO:tasks.workunit.client.0.smithi040.stdout:found image16 2024-09-11T14:36:15.536 INFO:tasks.workunit.client.0.smithi040.stdout:erased image16 2024-09-11T14:36:15.537 INFO:tasks.workunit.client.0.smithi040.stdout:image: image16 2024-09-11T14:36:15.537 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image16 2024-09-11T14:36:15.537 INFO:tasks.workunit.client.0.smithi040.stdout:found image16 2024-09-11T14:36:15.537 INFO:tasks.workunit.client.0.smithi040.stdout:erased image16 2024-09-11T14:36:15.537 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (4336 ms) 2024-09-11T14:36:15.537 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-09-11T14:36:17.038 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:19.240 INFO:tasks.workunit.client.0.smithi040.stdout:num images is: 1 2024-09-11T14:36:19.240 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:19.240 INFO:tasks.workunit.client.0.smithi040.stdout:image: image17 2024-09-11T14:36:19.240 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image17 2024-09-11T14:36:19.240 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:19.324 INFO:tasks.workunit.client.0.smithi040.stdout:num images is: 2 2024-09-11T14:36:19.324 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 2 2024-09-11T14:36:19.324 INFO:tasks.workunit.client.0.smithi040.stdout:image: image17 2024-09-11T14:36:19.324 INFO:tasks.workunit.client.0.smithi040.stdout:image: image18 2024-09-11T14:36:19.324 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image17 2024-09-11T14:36:19.324 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image18 2024-09-11T14:36:19.485 INFO:tasks.workunit.client.0.smithi040.stdout:num images is: 1 2024-09-11T14:36:19.486 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:19.486 INFO:tasks.workunit.client.0.smithi040.stdout:image: image18 2024-09-11T14:36:19.486 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image18 2024-09-11T14:36:19.486 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (3950 ms) 2024-09-11T14:36:19.486 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCopy 2024-09-11T14:36:21.059 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:30.216 INFO:tasks.workunit.client.0.smithi040.stdout:image: image19 2024-09-11T14:36:30.216 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image19 2024-09-11T14:36:30.216 INFO:tasks.workunit.client.0.smithi040.stdout:found image19 2024-09-11T14:36:30.216 INFO:tasks.workunit.client.0.smithi040.stdout:erased image19 2024-09-11T14:36:30.216 INFO:tasks.workunit.client.0.smithi040.stdout:image: image19 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:image: image20 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image19 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:found image19 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:erased image19 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image20 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:found image20 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:erased image20 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:0.00% done 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:100.00% done 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:image: image19 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:image: image20 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:image: image21 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image19 2024-09-11T14:36:30.217 INFO:tasks.workunit.client.0.smithi040.stdout:found image19 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:erased image19 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image20 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:found image20 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:erased image20 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image21 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:found image21 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:erased image21 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCopy (10730 ms) 2024-09-11T14:36:30.218 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-09-11T14:36:32.098 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:num images is: 1 2024-09-11T14:36:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:image: image22 2024-09-11T14:36:37.629 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image22 2024-09-11T14:36:39.622 INFO:tasks.workunit.client.0.smithi040.stdout:num images is: 2 2024-09-11T14:36:39.622 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 2 2024-09-11T14:36:39.622 INFO:tasks.workunit.client.0.smithi040.stdout:image: image22 2024-09-11T14:36:39.622 INFO:tasks.workunit.client.0.smithi040.stdout:image: image23 2024-09-11T14:36:39.622 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image22 2024-09-11T14:36:39.622 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image23 2024-09-11T14:36:40.530 INFO:tasks.workunit.client.0.smithi040.stdout:0.00% done 2024-09-11T14:36:40.530 INFO:tasks.workunit.client.0.smithi040.stdout:100.00% done 2024-09-11T14:36:40.530 INFO:tasks.workunit.client.0.smithi040.stdout:num images is: 3 2024-09-11T14:36:40.530 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 3 2024-09-11T14:36:40.530 INFO:tasks.workunit.client.0.smithi040.stdout:image: image22 2024-09-11T14:36:40.530 INFO:tasks.workunit.client.0.smithi040.stdout:image: image23 2024-09-11T14:36:40.531 INFO:tasks.workunit.client.0.smithi040.stdout:image: image24 2024-09-11T14:36:40.531 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image22 2024-09-11T14:36:40.531 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image23 2024-09-11T14:36:40.531 INFO:tasks.workunit.client.0.smithi040.stdout:expected = image24 2024-09-11T14:36:41.260 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCopyPP (11044 ms) 2024-09-11T14:36:41.260 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-09-11T14:36:41.260 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:1578: Skipped 2024-09-11T14:36:41.261 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:36:41.261 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy (0 ms) 2024-09-11T14:36:41.261 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-09-11T14:36:41.261 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:1745: Skipped 2024-09-11T14:36:41.261 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:36:41.261 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP (0 ms) 2024-09-11T14:36:41.261 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-09-11T14:36:41.261 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:45.413 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 1 2024-09-11T14:36:45.413 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap1 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:found snap1 with size 2097152 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 2 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 2 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap1 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap2 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:found snap1 with size 2097152 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:found snap2 with size 4194304 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 1 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap2 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:found snap2 with size 4194304 2024-09-11T14:36:45.414 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 0 2024-09-11T14:36:45.415 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 0 2024-09-11T14:36:45.415 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (4153 ms) 2024-09-11T14:36:45.415 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-09-11T14:36:45.415 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:1890: Skipped 2024-09-11T14:36:45.415 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:36:45.415 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp (0 ms) 2024-09-11T14:36:45.415 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-09-11T14:36:45.415 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:46.345 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 1 2024-09-11T14:36:46.345 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:46.345 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap1 2024-09-11T14:36:46.345 INFO:tasks.workunit.client.0.smithi040.stdout:found snap1 with size 2097152 2024-09-11T14:36:47.393 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 2 2024-09-11T14:36:47.393 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 2 2024-09-11T14:36:47.393 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap1 2024-09-11T14:36:47.393 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap2 2024-09-11T14:36:47.393 INFO:tasks.workunit.client.0.smithi040.stdout:found snap1 with size 2097152 2024-09-11T14:36:47.394 INFO:tasks.workunit.client.0.smithi040.stdout:found snap2 with size 4194304 2024-09-11T14:36:48.406 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 1 2024-09-11T14:36:48.406 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:48.406 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap2 2024-09-11T14:36:48.406 INFO:tasks.workunit.client.0.smithi040.stdout:found snap2 with size 4194304 2024-09-11T14:36:49.406 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 0 2024-09-11T14:36:49.406 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 0 2024-09-11T14:36:49.425 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4012 ms) 2024-09-11T14:36:49.426 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-09-11T14:36:49.426 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:55.443 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 0 2024-09-11T14:36:55.443 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 0 2024-09-11T14:36:55.468 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6043 ms) 2024-09-11T14:36:55.468 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-09-11T14:36:55.468 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:36:56.414 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 1 2024-09-11T14:36:56.414 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:56.414 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap1 2024-09-11T14:36:56.414 INFO:tasks.workunit.client.0.smithi040.stdout:found snap1 with size 2097152 2024-09-11T14:36:57.409 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 2 2024-09-11T14:36:57.410 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 2 2024-09-11T14:36:57.410 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap1 2024-09-11T14:36:57.410 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap2 2024-09-11T14:36:57.410 INFO:tasks.workunit.client.0.smithi040.stdout:found snap1 with size 2097152 2024-09-11T14:36:57.410 INFO:tasks.workunit.client.0.smithi040.stdout:found snap2 with size 4194304 2024-09-11T14:36:57.477 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 2 2024-09-11T14:36:57.478 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 2 2024-09-11T14:36:57.478 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap1-rename 2024-09-11T14:36:57.478 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap2 2024-09-11T14:36:57.478 INFO:tasks.workunit.client.0.smithi040.stdout:found snap1-rename with size 2097152 2024-09-11T14:36:57.478 INFO:tasks.workunit.client.0.smithi040.stdout:found snap2 with size 4194304 2024-09-11T14:36:58.498 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 1 2024-09-11T14:36:58.498 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:36:58.498 INFO:tasks.workunit.client.0.smithi040.stdout:snap: snap2-rename 2024-09-11T14:36:58.498 INFO:tasks.workunit.client.0.smithi040.stdout:found snap2-rename with size 4194304 2024-09-11T14:36:59.444 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 0 2024-09-11T14:36:59.444 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 0 2024-09-11T14:36:59.464 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (3996 ms) 2024-09-11T14:36:59.464 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-09-11T14:37:01.220 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:37:01.220 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:37:01.220 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:37:01.220 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:37:05.113 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (5649 ms) 2024-09-11T14:37:05.113 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-09-11T14:37:09.566 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:37:32.740 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.RemoveFullTry (27626 ms) 2024-09-11T14:37:32.740 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2227: Skipped 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool (0 ms) 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2247: Skipped 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespace (0 ms) 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2269: Skipped 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespaceDataPool (0 ms) 2024-09-11T14:37:32.741 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestIO 2024-09-11T14:37:32.742 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:38:05.740 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.741 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:38:05.742 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:38:05.743 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.744 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.745 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:38:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:discard: 512~512 = 512 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:aio discard: 1536~512 = 0 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.747 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.748 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:38:05.749 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:38:05.750 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.751 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.752 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.754 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:05.755 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:05.756 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:05.756 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:05.756 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:05.756 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:05.756 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:05.756 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.630 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.631 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.632 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.633 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.634 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.635 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.636 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.637 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:38:33.638 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:38:33.639 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:38:33.639 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:38:33.639 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:38:33.639 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:38:33.639 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestIO (60890 ms) 2024-09-11T14:38:33.639 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-09-11T14:38:33.639 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:39:11.791 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.792 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:11.793 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:39:11.794 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.795 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.796 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:39:11.797 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:discard: 512~512 = 512 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:aio discard: 1536~512 = 0 2024-09-11T14:39:11.798 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:39:11.799 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.800 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.801 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.802 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.803 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.804 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:11.805 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.806 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:11.807 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:11.808 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.505 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.506 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.507 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.508 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.509 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.510 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.511 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.512 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.513 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:39:46.514 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 4 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:read: 4 2024-09-11T14:39:46.515 INFO:tasks.workunit.client.0.smithi040.stdout:read: 4 2024-09-11T14:39:46.516 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (72875 ms) 2024-09-11T14:39:46.516 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-09-11T14:39:46.516 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:39:53.566 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:53.566 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:53.566 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:53.566 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:53.566 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:39:53.566 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:53.566 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:53.566 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:53.567 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:39:53.568 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: -84 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:495: Failure 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout:Expected equality of these values: 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout: len 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout: Which is: 512 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout: static_cast(written) 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout: Which is: 18446744073709551532 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:630: Failure 2024-09-11T14:39:53.569 INFO:tasks.workunit.client.0.smithi040.stdout:Value of: passed 2024-09-11T14:39:53.570 INFO:tasks.workunit.client.0.smithi040.stdout: Actual: false 2024-09-11T14:39:53.570 INFO:tasks.workunit.client.0.smithi040.stdout:Expected: true 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 4 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:read: 4 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:read: 4 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 (12168 ms) 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2471: Skipped 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption (0 ms) 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2659: Skipped 2024-09-11T14:39:58.673 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:39:58.674 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (0 ms) 2024-09-11T14:39:58.674 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-09-11T14:39:58.674 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2735: Skipped 2024-09-11T14:39:58.674 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:39:58.674 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (0 ms) 2024-09-11T14:39:58.674 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-09-11T14:39:58.674 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:40:03.281 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4608 ms) 2024-09-11T14:40:03.281 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-09-11T14:40:03.281 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:40:09.592 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (6311 ms) 2024-09-11T14:40:09.592 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-11T14:40:09.592 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2905: Skipped 2024-09-11T14:40:09.592 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:40:09.592 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2024-09-11T14:40:09.592 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-09-11T14:40:09.592 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2985: Skipped 2024-09-11T14:40:09.592 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:40:09.593 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch (0 ms) 2024-09-11T14:40:09.593 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-09-11T14:40:09.593 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:40:22.519 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.EncryptedResize (12927 ms) 2024-09-11T14:40:22.519 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-09-11T14:40:22.519 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:3202: Skipped 2024-09-11T14:40:22.519 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:40:22.519 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData (0 ms) 2024-09-11T14:40:22.519 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-09-11T14:40:22.519 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:40:54.997 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:54.998 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:40:54.999 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:40:55.000 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.002 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:40:55.003 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:discard: 512~512 = 512 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:aio discard: 1536~512 = 0 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.004 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.005 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.006 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.007 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.008 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.010 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.011 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.012 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:40:55.013 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:40:55.013 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:40:55.013 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:40:55.013 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:40:55.013 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:40:55.013 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:40:55.013 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.835 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.835 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.835 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.835 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.835 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.836 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.837 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.838 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.839 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.840 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.841 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.842 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.843 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.843 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.843 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.843 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.843 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.843 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.843 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.844 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.844 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.844 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.844 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.844 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.845 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.845 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.845 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.845 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.845 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.846 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:22.846 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:22.846 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:41:22.846 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:41:22.846 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:41:22.846 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.846 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (60316 ms) 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-09-11T14:41:22.847 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:3947: Skipped 2024-09-11T14:41:22.848 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:41:22.848 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO (0 ms) 2024-09-11T14:41:22.848 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-09-11T14:41:22.848 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:23.045 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (210 ms) 2024-09-11T14:41:23.045 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-09-11T14:41:23.045 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:23.292 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (247 ms) 2024-09-11T14:41:23.292 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-09-11T14:41:23.292 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:23.500 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (208 ms) 2024-09-11T14:41:23.501 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-09-11T14:41:23.501 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:23.693 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (193 ms) 2024-09-11T14:41:23.694 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-09-11T14:41:23.694 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:24.433 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (739 ms) 2024-09-11T14:41:24.433 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-09-11T14:41:24.433 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:24.989 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (556 ms) 2024-09-11T14:41:24.989 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-09-11T14:41:24.989 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:25.827 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (838 ms) 2024-09-11T14:41:25.827 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-09-11T14:41:25.827 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:26.748 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (922 ms) 2024-09-11T14:41:26.748 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-09-11T14:41:26.748 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:31.494 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (4746 ms) 2024-09-11T14:41:31.495 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-09-11T14:41:31.495 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:36.327 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (4833 ms) 2024-09-11T14:41:36.328 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-09-11T14:41:36.328 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:36.536 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (209 ms) 2024-09-11T14:41:36.537 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-09-11T14:41:36.537 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:36.765 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (229 ms) 2024-09-11T14:41:36.765 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-09-11T14:41:36.765 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:36.989 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (224 ms) 2024-09-11T14:41:36.989 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-09-11T14:41:36.990 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:37.174 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (185 ms) 2024-09-11T14:41:37.175 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-09-11T14:41:37.175 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:37.340 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:37.340 INFO:tasks.workunit.client.0.smithi040.stdout:aio discard: 0~1048576 = 0 2024-09-11T14:41:37.340 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:37.340 INFO:tasks.workunit.client.0.smithi040.stdout:aio discard: 0~4194304 = 0 2024-09-11T14:41:37.340 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:37.341 INFO:tasks.workunit.client.0.smithi040.stdout:aio discard: 3145728~1048576 = 0 2024-09-11T14:41:37.341 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (165 ms) 2024-09-11T14:41:37.341 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestFUA 2024-09-11T14:41:37.341 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.033 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.034 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.035 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.036 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.037 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestFUA (692 ms) 2024-09-11T14:41:38.037 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-09-11T14:41:38.037 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:41:38.142 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.142 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.143 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:41:38.144 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.465 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.465 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:start compare and write 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:compare and wrote: 512 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.466 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:41:38.467 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.489 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.489 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:41:38.489 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.489 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:41:38.489 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.512 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.512 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:41:38.512 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.512 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:41:38.512 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.535 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.535 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:41:38.535 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.535 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:41:38.535 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.559 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.559 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:41:38.559 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.559 INFO:tasks.workunit.client.0.smithi040.stdout:started aio compare and write 2024-09-11T14:41:38.559 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:41:38.606 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:41:38.606 INFO:tasks.workunit.client.0.smithi040.stdout:finished aio compare and write 2024-09-11T14:41:38.606 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.606 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.606 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.607 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.607 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:41:38.607 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.607 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.607 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.612 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.612 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:41:38.612 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.612 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.612 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.620 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.620 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:41:38.620 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.620 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.620 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.627 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.627 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:41:38.627 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.627 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.627 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.633 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.634 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:41:38.634 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:41:38.634 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:41:38.634 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:41:38.634 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:41:38.634 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:41:38.634 INFO:tasks.workunit.client.0.smithi040.stdout:discard: 512~512 2024-09-11T14:41:38.634 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:02.826 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:42:02.827 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:42:02.828 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:42:02.829 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:42:02.830 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.831 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.832 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:02.833 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:03.081 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:03.081 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:03.081 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:03.081 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:03.081 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:03.081 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:03.081 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:03.244 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:03.244 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:03.244 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:03.244 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:03.244 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:03.244 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:03.244 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:03.245 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:03.245 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:03.245 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:03.245 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:03.245 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:03.245 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:03.911 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:03.912 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:03.912 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:03.912 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:03.912 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:03.912 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:03.912 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:04.346 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:04.346 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:04.346 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:04.346 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:04.347 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:04.347 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:04.347 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:05.227 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:05.227 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:05.227 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:05.227 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:05.227 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:05.227 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:05.227 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:05.872 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:05.873 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:05.873 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:05.873 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:07.205 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:07.205 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:07.205 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:07.205 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:07.205 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:07.205 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:07.205 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:08.099 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:08.100 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:08.100 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:08.100 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:08.100 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:08.100 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:08.100 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:09.687 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:09.687 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:09.687 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:09.687 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:09.687 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:09.687 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:09.688 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:10.687 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:10.688 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:12.656 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:12.656 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:12.656 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:12.656 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:12.656 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:12.656 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:12.656 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:14.011 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:14.011 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:14.011 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:14.011 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:14.011 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:14.011 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:14.011 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:16.242 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:16.243 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:16.243 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:16.243 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:16.243 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:16.243 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:16.243 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:17.715 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:17.715 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:17.715 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:17.715 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:17.715 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:17.715 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:17.715 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:17.715 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:17.716 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:17.716 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:17.716 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:17.716 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:17.716 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:20.349 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:20.350 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:20.350 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:20.350 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:20.350 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:20.350 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:20.350 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:22.108 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:22.109 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:22.109 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:22.109 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:22.109 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:22.109 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:22.109 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:25.011 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:25.011 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:25.011 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:25.011 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:25.011 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:25.011 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:25.011 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:26.906 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestIOPP (48875 ms) 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-09-11T14:42:26.907 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:27.097 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (191 ms) 2024-09-11T14:42:27.098 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-09-11T14:42:27.098 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:27.294 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (197 ms) 2024-09-11T14:42:27.295 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-09-11T14:42:27.295 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (168 ms) 2024-09-11T14:42:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-09-11T14:42:27.463 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:27.620 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (157 ms) 2024-09-11T14:42:27.620 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-09-11T14:42:27.620 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:27.831 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (212 ms) 2024-09-11T14:42:27.832 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-09-11T14:42:27.832 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:27.983 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:28.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (191 ms) 2024-09-11T14:42:28.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-11T14:42:28.022 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:28.211 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (189 ms) 2024-09-11T14:42:28.212 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-11T14:42:28.212 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:28.360 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:28.398 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (187 ms) 2024-09-11T14:42:28.399 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-09-11T14:42:28.399 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:28.596 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (197 ms) 2024-09-11T14:42:28.596 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-09-11T14:42:28.596 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:28.761 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:28.798 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (202 ms) 2024-09-11T14:42:28.798 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-11T14:42:28.798 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:29.030 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (232 ms) 2024-09-11T14:42:29.030 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-11T14:42:29.030 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:29.221 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:29.264 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (234 ms) 2024-09-11T14:42:29.264 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-09-11T14:42:29.264 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:30.123 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (860 ms) 2024-09-11T14:42:30.124 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-09-11T14:42:30.124 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:30.621 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:31.004 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (880 ms) 2024-09-11T14:42:31.004 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-09-11T14:42:31.004 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:31.988 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (984 ms) 2024-09-11T14:42:31.988 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-09-11T14:42:31.988 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:32.265 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:32.563 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (576 ms) 2024-09-11T14:42:32.563 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-09-11T14:42:32.563 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:37.185 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (4621 ms) 2024-09-11T14:42:37.186 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-09-11T14:42:37.186 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:41.836 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:42.311 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (5127 ms) 2024-09-11T14:42:42.311 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-09-11T14:42:42.311 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:42:42.562 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:42:42.562 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:42:42.562 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:42:42.562 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:42:42.563 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:42:42.563 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.563 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:42:42.563 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:42.583 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:42.584 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:42:42.584 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.584 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:42:42.584 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:42.605 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:42.605 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:42:42.605 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.605 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:42:42.605 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:42.627 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:42.627 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:42:42.627 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.627 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:42:42.627 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:42.648 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:42.649 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:42:42.649 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.649 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:42:42.649 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:42:42.666 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:42:42.666 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:42:42.666 INFO:tasks.workunit.client.0.smithi040.stdout:read: 512 2024-09-11T14:42:42.666 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.666 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:42:42.666 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:42:42.679 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:42:42.679 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:42:42.679 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.679 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:42:42.679 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:42:42.687 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:42:42.687 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:42:42.687 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.688 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:42:42.688 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:42:42.697 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:42:42.698 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:42:42.698 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.698 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:42:42.698 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:42:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:42:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:42:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:42:42.705 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:42:42.706 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:43:07.253 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:43:07.253 INFO:tasks.workunit.client.0.smithi040.stdout:finished read 2024-09-11T14:43:07.253 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:43:07.253 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.253 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.253 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 0 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 16384 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.254 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 49152 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 65536 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.255 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 98304 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 114688 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.256 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.257 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.257 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.257 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:43:07.257 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.257 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.257 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 147456 2024-09-11T14:43:07.257 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 163840 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 196608 2024-09-11T14:43:07.258 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 212992 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:07.259 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:07.260 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.260 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.260 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:07.260 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:07.260 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:07.272 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:07.272 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:07.272 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.272 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.272 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:07.272 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:07.273 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:07.508 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:07.509 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:07.509 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.509 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.509 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:07.509 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:07.509 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:07.669 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:07.670 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:07.670 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:07.670 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:07.670 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:07.670 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:07.670 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.670 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:07.671 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:07.671 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:07.671 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:07.671 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:07.671 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:08.335 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:08.335 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:08.335 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:08.335 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:08.335 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:08.336 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:08.336 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:08.790 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:08.790 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:08.790 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:08.790 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:08.790 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:08.790 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:08.790 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:09.735 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:09.735 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:09.735 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:09.735 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:09.735 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:09.735 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:09.735 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:10.334 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:10.334 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:10.334 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:10.334 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:10.334 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:10.334 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:10.334 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:10.334 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:10.335 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:10.335 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:10.335 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:10.335 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:10.335 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:11.705 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:11.705 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:11.705 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:11.705 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:11.705 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:11.705 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:11.705 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:12.607 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:12.607 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:12.607 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:12.607 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:12.607 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:12.607 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:12.607 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:14.190 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:14.190 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:14.191 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:14.191 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:14.191 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:14.191 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:14.191 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:15.233 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:15.233 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:15.233 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:15.233 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:15.233 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:15.233 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:15.233 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:15.234 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:15.234 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:15.234 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:15.234 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:15.234 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:15.234 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:17.190 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:17.191 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:17.191 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:17.191 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:17.191 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:17.191 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:17.191 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:18.549 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:18.549 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:18.549 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:18.549 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:18.549 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:18.550 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:18.550 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:20.825 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:20.825 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:20.825 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:20.825 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:20.825 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:20.825 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:20.825 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:22.281 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:22.282 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:22.283 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:25.021 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:25.022 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:25.022 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:25.022 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:25.022 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:25.022 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:25.022 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:26.799 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:26.799 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:26.799 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:26.799 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:26.799 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:26.799 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:26.799 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:29.761 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:29.761 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:29.761 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:29.761 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:29.762 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:29.762 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:29.762 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:43:31.649 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:finished writesame 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:to verify the data 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:verified 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:started writesame 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:expected fail, finished writesame 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (49338 ms) 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-09-11T14:43:31.650 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 0 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 0 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 1 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:snap: orig 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:found orig with size 2097152 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:read: 80 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:write test data! 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 80 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 2 2024-09-11T14:43:33.587 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 2 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:snap: orig 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:snap: written 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:found orig with size 2097152 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:found written with size 2097152 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:read: 80 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:read: 80 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:read: 80 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:write to snapshot returned -30 2024-09-11T14:43:33.588 INFO:tasks.workunit.client.0.smithi040.stdout:Read-only file system 2024-09-11T14:43:33.786 INFO:tasks.workunit.client.0.smithi040.stdout:read: 80 2024-09-11T14:43:33.786 INFO:tasks.workunit.client.0.smithi040.stdout:read: 80 2024-09-11T14:43:33.786 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 80 2024-09-11T14:43:33.787 INFO:tasks.workunit.client.0.smithi040.stdout:opening testimg@orig 2024-09-11T14:43:33.787 INFO:tasks.workunit.client.0.smithi040.stdout:read: 80 2024-09-11T14:43:33.787 INFO:tasks.workunit.client.0.smithi040.stdout:write to snapshot returned -30 2024-09-11T14:43:33.787 INFO:tasks.workunit.client.0.smithi040.stdout:Read-only file system 2024-09-11T14:43:35.538 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 2 2024-09-11T14:43:35.538 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 2 2024-09-11T14:43:35.538 INFO:tasks.workunit.client.0.smithi040.stdout:snap: orig 2024-09-11T14:43:35.538 INFO:tasks.workunit.client.0.smithi040.stdout:snap: written 2024-09-11T14:43:35.538 INFO:tasks.workunit.client.0.smithi040.stdout:found orig with size 2097152 2024-09-11T14:43:35.538 INFO:tasks.workunit.client.0.smithi040.stdout:found written with size 2097152 2024-09-11T14:43:35.538 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 1 2024-09-11T14:43:35.538 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 1 2024-09-11T14:43:35.539 INFO:tasks.workunit.client.0.smithi040.stdout:snap: orig 2024-09-11T14:43:35.539 INFO:tasks.workunit.client.0.smithi040.stdout:found orig with size 2097152 2024-09-11T14:43:35.539 INFO:tasks.workunit.client.0.smithi040.stdout:num snaps is: 0 2024-09-11T14:43:35.539 INFO:tasks.workunit.client.0.smithi040.stdout:expected: 0 2024-09-11T14:43:35.539 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (3888 ms) 2024-09-11T14:43:35.539 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-09-11T14:43:35.539 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:37.486 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (1948 ms) 2024-09-11T14:43:37.486 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestClone 2024-09-11T14:43:37.486 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:6444: Skipped 2024-09-11T14:43:37.486 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:43:37.486 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestClone (0 ms) 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestClone2 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:6608: Skipped 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestClone2 (0 ms) 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ListChildren 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:6841: Skipped 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ListChildren (0 ms) 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:6995: Skipped 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:Not supported by crimson yet. Skipped 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered (0 ms) 2024-09-11T14:43:37.487 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.LockingPP 2024-09-11T14:43:37.488 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:38.103 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.LockingPP (617 ms) 2024-09-11T14:43:38.103 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.FlushAio 2024-09-11T14:43:38.103 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:43.714 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.FlushAio (5612 ms) 2024-09-11T14:43:43.715 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-09-11T14:43:43.715 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:48.750 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.FlushAioPP (5035 ms) 2024-09-11T14:43:48.750 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-09-11T14:43:48.750 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:48.925 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (175 ms) 2024-09-11T14:43:48.925 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-09-11T14:43:48.925 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:49.112 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (188 ms) 2024-09-11T14:43:49.113 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-09-11T14:43:49.113 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:49.256 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.ZeroLengthRead (143 ms) 2024-09-11T14:43:49.256 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-09-11T14:43:49.256 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:49.714 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.LargeCacheRead (458 ms) 2024-09-11T14:43:49.714 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9152: Skipped 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestPendingAio (0 ms) 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9264: Skipped 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyup (0 ms) 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9325: Skipped 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:43:49.715 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyupMismatch (0 ms) 2024-09-11T14:43:49.716 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.Flatten 2024-09-11T14:43:49.716 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9336: Skipped 2024-09-11T14:43:49.716 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:43:49.716 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.Flatten (0 ms) 2024-09-11T14:43:49.716 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.Sparsify 2024-09-11T14:43:49.716 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:49.990 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 32768 2024-09-11T14:43:49.990 INFO:tasks.workunit.client.0.smithi040.stdout:read: 32768 2024-09-11T14:43:49.991 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.Sparsify (277 ms) 2024-09-11T14:43:49.991 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-09-11T14:43:49.991 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:50.244 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.SparsifyPP (254 ms) 2024-09-11T14:43:50.245 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-09-11T14:43:50.245 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:43:55.507 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.SnapshotLimit (5262 ms) 2024-09-11T14:43:55.507 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-09-11T14:43:55.507 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:00.572 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5063 ms) 2024-09-11T14:44:00.572 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-09-11T14:44:00.572 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9523: Skipped 2024-09-11T14:44:00.572 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.572 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RebuildObjectMapViaLockOwner (0 ms) 2024-09-11T14:44:00.572 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-09-11T14:44:00.572 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9575: Skipped 2024-09-11T14:44:00.572 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RenameViaLockOwner (1 ms) 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9615: Skipped 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapCreateViaLockOwner (0 ms) 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9659: Skipped 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveViaLockOwner (0 ms) 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9699: Skipped 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.573 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.UpdateFeaturesViaLockOwner (0 ms) 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9736: Skipped 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9777: Skipped 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapRemove2 (0 ms) 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9823: Skipped 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapRenameViaLockOwner (0 ms) 2024-09-11T14:44:00.574 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9864: Skipped 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapProtectViaLockOwner (0 ms) 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9905: Skipped 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapUnprotectViaLockOwner (0 ms) 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9949: Skipped 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.FlattenViaLockOwner (0 ms) 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9997: Skipped 2024-09-11T14:44:00.575 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ResizeViaLockOwner (0 ms) 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10033: Skipped 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SparsifyViaLockOwner (0 ms) 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10069: Skipped 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ObjectMapConsistentSnap (0 ms) 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.Metadata 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10121: Skipped 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.576 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.Metadata (0 ms) 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10310: Skipped 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.MetadataPP (0 ms) 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10391: Skipped 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.UpdateFeatures (0 ms) 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-09-11T14:44:00.577 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.RebuildObjectMap (151 ms) 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10584: Skipped 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RebuildNewObjectMap (0 ms) 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10609: Skipped 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.722 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CheckObjectMap (0 ms) 2024-09-11T14:44:00.723 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-09-11T14:44:00.723 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:00.905 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.BlockingAIO (182 ms) 2024-09-11T14:44:00.905 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-09-11T14:44:00.905 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10737: Skipped 2024-09-11T14:44:00.905 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.905 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockTransition (0 ms) 2024-09-11T14:44:00.906 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-09-11T14:44:00.906 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10793: Skipped 2024-09-11T14:44:00.906 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.906 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-09-11T14:44:00.906 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-09-11T14:44:00.906 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10846: Skipped 2024-09-11T14:44:00.907 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:00.907 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CacheMayCopyOnWrite (0 ms) 2024-09-11T14:44:00.907 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-09-11T14:44:00.907 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:01.644 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (740 ms) 2024-09-11T14:44:01.644 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-09-11T14:44:06.960 INFO:tasks.workunit.client.0.smithi040.stdout:0.00% done 2024-09-11T14:44:06.961 INFO:tasks.workunit.client.0.smithi040.stdout:100.00% done 2024-09-11T14:44:06.961 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestImageOptions (5317 ms) 2024-09-11T14:44:06.961 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-09-11T14:44:08.995 INFO:tasks.workunit.client.0.smithi040.stdout:0.00% done 2024-09-11T14:44:08.995 INFO:tasks.workunit.client.0.smithi040.stdout:100.00% done 2024-09-11T14:44:08.996 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (2034 ms) 2024-09-11T14:44:08.996 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-09-11T14:44:08.996 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2024-09-11T14:44:08.996 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-09-11T14:44:08.996 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-09-11T14:44:08.996 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-09-11T14:44:08.996 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:09.395 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:44:09.395 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:44:09.395 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:44:09.395 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:44:09.395 INFO:tasks.workunit.client.0.smithi040.stdout:wrote: 512 2024-09-11T14:44:09.395 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.395 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:44:09.396 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:started write 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:write completion cb called! 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 0 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:finished write 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:44:09.397 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:44:09.398 INFO:tasks.workunit.client.0.smithi040.stdout:created completion 2024-09-11T14:44:09.399 INFO:tasks.workunit.client.0.smithi040.stdout:started read 2024-09-11T14:44:09.399 INFO:tasks.workunit.client.0.smithi040.stdout:read completion cb called! 2024-09-11T14:44:09.399 INFO:tasks.workunit.client.0.smithi040.stdout:return value is: 512 2024-09-11T14:44:09.399 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.ImagePollIO (400 ms) 2024-09-11T14:44:09.399 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.Mirror 2024-09-11T14:44:09.479 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:09.588 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:10.062 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.Mirror (667 ms) 2024-09-11T14:44:10.062 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-09-11T14:44:10.229 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (167 ms) 2024-09-11T14:44:10.229 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-09-11T14:44:10.229 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11322: Skipped 2024-09-11T14:44:10.229 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:10.229 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CreateWithMirrorEnabled (0 ms) 2024-09-11T14:44:10.229 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-09-11T14:44:10.229 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11364: Skipped 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (0 ms) 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11410: Skipped 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLock (0 ms) 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.BreakLock 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11553: Skipped 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:Not supported by crimson yet. Skipped 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.BreakLock (0 ms) 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-09-11T14:44:10.230 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:10.356 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (127 ms) 2024-09-11T14:44:10.356 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11676: Skipped 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurge (0 ms) 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11722: Skipped 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (0 ms) 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11753: Skipped 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:10.357 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestTrashPurge (0 ms) 2024-09-11T14:44:10.358 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-09-11T14:44:10.358 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11810: Skipped 2024-09-11T14:44:10.358 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:10.358 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndRestore (0 ms) 2024-09-11T14:44:10.358 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-09-11T14:44:10.358 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:10.594 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestListWatchers (239 ms) 2024-09-11T14:44:10.595 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-09-11T14:44:10.595 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:11.514 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.TestSetSnapById (920 ms) 2024-09-11T14:44:11.514 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.Namespaces 2024-09-11T14:44:11.898 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.Namespaces (384 ms) 2024-09-11T14:44:11.898 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-09-11T14:44:12.375 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:12.390 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.NamespacesPP (492 ms) 2024-09-11T14:44:12.390 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.Migration 2024-09-11T14:44:12.390 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:12.390 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:17.323 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.Migration (4932 ms) 2024-09-11T14:44:17.323 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-09-11T14:44:17.323 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:17.323 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:22.360 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.MigrationPP (5037 ms) 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12177: Skipped 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestGetAccessTimestamp (0 ms) 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12202: Skipped 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestGetModifyTimestamp (0 ms) 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12225: Skipped 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ZeroOverlapFlatten (0 ms) 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-09-11T14:44:22.361 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12258: Skipped 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.PoolMetadata (0 ms) 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12370: Skipped 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.PoolMetadataPP (0 ms) 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.Config 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12422: Skipped 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.Config (0 ms) 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12506: Skipped 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.362 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ConfigPP (0 ms) 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12583: Skipped 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.PoolStatsPP (0 ms) 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12658: Skipped 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ImageSpec (0 ms) 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:12853: Skipped 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveWithChildMissing (0 ms) 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-09-11T14:44:22.363 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:28.642 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.QuiesceWatch (6282 ms) 2024-09-11T14:44:28.643 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-09-11T14:44:28.643 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:34.640 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (5998 ms) 2024-09-11T14:44:34.640 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-09-11T14:44:34.640 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:13115: Skipped 2024-09-11T14:44:34.640 INFO:tasks.workunit.client.0.smithi040.stdout:Not supported by crimson yet. Skipped 2024-09-11T14:44:34.640 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError (0 ms) 2024-09-11T14:44:34.641 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-09-11T14:44:34.641 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:44:34.776 INFO:tasks.workunit.client.0.smithi040.stderr:test quiesce is not long enough to time out 2024-09-11T14:44:43.635 INFO:tasks.workunit.client.0.smithi040.stderr:test quiesce is timed out 2024-09-11T14:44:43.661 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 0 2024-09-11T14:44:44.661 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 1 2024-09-11T14:44:45.662 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 2 2024-09-11T14:44:46.662 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 3 2024-09-11T14:44:47.662 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 4 2024-09-11T14:44:48.662 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 5 2024-09-11T14:44:49.662 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 6 2024-09-11T14:44:50.662 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 7 2024-09-11T14:44:51.662 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 8 2024-09-11T14:44:52.663 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 9 2024-09-11T14:44:53.663 INFO:tasks.workunit.client.0.smithi040.stderr:waiting for timed out ... 10 2024-09-11T14:45:04.663 INFO:tasks.workunit.client.0.smithi040.stderr:test retry succeeds 2024-09-11T14:45:07.512 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (32872 ms) 2024-09-11T14:45:07.512 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-09-11T14:45:07.512 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:45:07.777 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.WriteZeroes (266 ms) 2024-09-11T14:45:07.777 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-09-11T14:45:07.778 INFO:tasks.workunit.client.0.smithi040.stdout:using old format 2024-09-11T14:45:08.174 INFO:tasks.workunit.client.0.smithi040.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (396 ms) 2024-09-11T14:45:08.174 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.ConcurrentOperations 2024-09-11T14:45:08.174 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:13474: Skipped 2024-09-11T14:45:08.174 INFO:tasks.workunit.client.0.smithi040.stdout:Not supported by crimson yet. Skipped 2024-09-11T14:45:08.174 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations (0 ms) 2024-09-11T14:45:08.174 INFO:tasks.workunit.client.0.smithi040.stdout:[ RUN ] TestLibRBD.FormatAndCloneFormatOptions 2024-09-11T14:45:08.175 INFO:tasks.workunit.client.0.smithi040.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:13630: Skipped 2024-09-11T14:45:08.175 INFO:tasks.workunit.client.0.smithi040.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:45:08.175 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.FormatAndCloneFormatOptions (0 ms) 2024-09-11T14:45:15.019 INFO:tasks.workunit.client.0.smithi040.stdout:[----------] 171 tests from TestLibRBD (542483 ms total) 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[----------] Global test environment tear-down 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[==========] 171 tests from 1 test suite ran. (549380 ms total) 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[ PASSED ] 103 tests. 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] 67 tests, listed below: 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP 2024-09-11T14:45:15.020 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespace 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestClone 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestClone2 2024-09-11T14:45:15.021 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ListChildren 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestPendingAio 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyup 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.Flatten 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RenameViaLockOwner 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapCreateViaLockOwner 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveViaLockOwner 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapRemove2 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapRenameViaLockOwner 2024-09-11T14:45:15.022 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapProtectViaLockOwner 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapUnprotectViaLockOwner 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.FlattenViaLockOwner 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ResizeViaLockOwner 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SparsifyViaLockOwner 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ObjectMapConsistentSnap 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.Metadata 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.MetadataPP 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.UpdateFeatures 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.RebuildNewObjectMap 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CheckObjectMap 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockTransition 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CacheMayCopyOnWrite 2024-09-11T14:45:15.023 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.CreateWithMirrorEnabled 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLock 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.BreakLock 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurge 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestTrashPurge 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndRestore 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestGetAccessTimestamp 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.TestGetModifyTimestamp 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ZeroOverlapFlatten 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.PoolMetadata 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.PoolMetadataPP 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.Config 2024-09-11T14:45:15.024 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ConfigPP 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.PoolStatsPP 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ImageSpec 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveWithChildMissing 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout:[ SKIPPED ] TestLibRBD.FormatAndCloneFormatOptions 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout:[ FAILED ] 1 test, listed below: 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout: 1 FAILED TEST 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout: YOU HAVE 1 DISABLED TEST 2024-09-11T14:45:15.025 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-09-11T14:45:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:45:15.033 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2024-09-11T14:45:15.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T14:45:15.595 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-09-11T14:45:15.842 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=34256468dda440d391fe1389507990e0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-09-11T14:45:15.844 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T14:45:15.854 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:45:15.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:15.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:15.977+0000 7ff009f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:15.993+0000 7ff009f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:16.218 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:45:16.219 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-09-11T14:45:16.235 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-11T14:45:15.925373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":371960,"kb_used_data":371960,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156521224,"statfs":{"total":160658620416,"available":160277733376,"internally_reserved":0,"allocated":380887040,"data_stored":380887040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168035816,"num_objects":-228,"num_object_clones":0,"num_object_copies":-456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28555,"num_read_kb":-141621,"num_write":-1614,"num_write_kb":-205412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001019"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.075066+0000","last_change":"2024-09-11T14:35:14.942992+0000","last_active":"2024-09-11T14:45:13.075066+0000","last_peered":"2024-09-11T14:45:13.075066+0000","last_clean":"2024-09-11T14:45:13.075066+0000","last_became_active":"2024-09-11T14:35:12.958460+0000","last_became_peered":"2024-09-11T14:35:12.958460+0000","last_unstale":"2024-09-11T14:45:13.075066+0000","last_undegraded":"2024-09-11T14:45:13.075066+0000","last_fullsized":"2024-09-11T14:45:13.075066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.112372+0000","last_change":"2024-09-11T14:35:14.946690+0000","last_active":"2024-09-11T14:45:13.112372+0000","last_peered":"2024-09-11T14:45:13.112372+0000","last_clean":"2024-09-11T14:45:13.112372+0000","last_became_active":"2024-09-11T14:35:13.015169+0000","last_became_peered":"2024-09-11T14:35:13.015169+0000","last_unstale":"2024-09-11T14:45:13.112372+0000","last_undegraded":"2024-09-11T14:45:13.112372+0000","last_fullsized":"2024-09-11T14:45:13.112372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.082880+0000","last_change":"2024-09-11T14:35:14.942753+0000","last_active":"2024-09-11T14:45:13.082880+0000","last_peered":"2024-09-11T14:45:13.082880+0000","last_clean":"2024-09-11T14:45:13.082880+0000","last_became_active":"2024-09-11T14:35:12.989419+0000","last_became_peered":"2024-09-11T14:35:12.989419+0000","last_unstale":"2024-09-11T14:45:13.082880+0000","last_undegraded":"2024-09-11T14:45:13.082880+0000","last_fullsized":"2024-09-11T14:45:13.082880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.073328+0000","last_change":"2024-09-11T14:35:14.947434+0000","last_active":"2024-09-11T14:45:13.073328+0000","last_peered":"2024-09-11T14:45:13.073328+0000","last_clean":"2024-09-11T14:45:13.073328+0000","last_became_active":"2024-09-11T14:35:12.976308+0000","last_became_peered":"2024-09-11T14:35:12.976308+0000","last_unstale":"2024-09-11T14:45:13.073328+0000","last_undegraded":"2024-09-11T14:45:13.073328+0000","last_fullsized":"2024-09-11T14:45:13.073328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":323,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-11T14:45:11.126639+0000","last_change":"2024-09-11T14:35:14.951771+0000","last_active":"2024-09-11T14:45:11.126639+0000","last_peered":"2024-09-11T14:45:11.126639+0000","last_clean":"2024-09-11T14:45:11.126639+0000","last_became_active":"2024-09-11T14:35:12.981976+0000","last_became_peered":"2024-09-11T14:35:12.981976+0000","last_unstale":"2024-09-11T14:45:11.126639+0000","last_undegraded":"2024-09-11T14:45:11.126639+0000","last_fullsized":"2024-09-11T14:45:11.126639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.151237+0000","last_change":"2024-09-11T14:35:14.951268+0000","last_active":"2024-09-11T14:45:13.151237+0000","last_peered":"2024-09-11T14:45:13.151237+0000","last_clean":"2024-09-11T14:45:13.151237+0000","last_became_active":"2024-09-11T14:35:12.945557+0000","last_became_peered":"2024-09-11T14:35:12.945557+0000","last_unstale":"2024-09-11T14:45:13.151237+0000","last_undegraded":"2024-09-11T14:45:13.151237+0000","last_fullsized":"2024-09-11T14:45:13.151237+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.153611+0000","last_change":"2024-09-11T14:35:14.944204+0000","last_active":"2024-09-11T14:45:13.153611+0000","last_peered":"2024-09-11T14:45:13.153611+0000","last_clean":"2024-09-11T14:45:13.153611+0000","last_became_active":"2024-09-11T14:35:12.965045+0000","last_became_peered":"2024-09-11T14:35:12.965045+0000","last_unstale":"2024-09-11T14:45:13.153611+0000","last_undegraded":"2024-09-11T14:45:13.153611+0000","last_fullsized":"2024-09-11T14:45:13.153611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":328,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.089773+0000","last_change":"2024-09-11T14:35:14.942049+0000","last_active":"2024-09-11T14:45:13.089773+0000","last_peered":"2024-09-11T14:45:13.089773+0000","last_clean":"2024-09-11T14:45:13.089773+0000","last_became_active":"2024-09-11T14:35:12.944374+0000","last_became_peered":"2024-09-11T14:35:12.944374+0000","last_unstale":"2024-09-11T14:45:13.089773+0000","last_undegraded":"2024-09-11T14:45:13.089773+0000","last_fullsized":"2024-09-11T14:45:13.089773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":399,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.112179+0000","last_change":"2024-09-11T14:35:10.963766+0000","last_active":"2024-09-11T14:45:13.112179+0000","last_peered":"2024-09-11T14:45:13.112179+0000","last_clean":"2024-09-11T14:45:13.112179+0000","last_became_active":"2024-09-11T14:35:10.942660+0000","last_became_peered":"2024-09-11T14:35:10.942660+0000","last_unstale":"2024-09-11T14:45:13.112179+0000","last_undegraded":"2024-09-11T14:45:13.112179+0000","last_fullsized":"2024-09-11T14:45:13.112179+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_clean_scrub_stamp":"2024-09-11T14:35:09.830264+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160281124864,"internally_reserved":0,"allocated":377495552,"data_stored":377495552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":80684,"kb_used_data":80684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52217044,"statfs":{"total":53552873472,"available":53470253056,"internally_reserved":0,"allocated":82620416,"data_stored":82620416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":157332,"kb_used_data":157332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52140396,"statfs":{"total":53552873472,"available":53391765504,"internally_reserved":0,"allocated":161107968,"data_stored":161107968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":133944,"kb_used_data":133944,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52163784,"statfs":{"total":53552873472,"available":53415714816,"internally_reserved":0,"allocated":137158656,"data_stored":137158656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53416517632,"internally_reserved":0,"allocated":136355840,"data_stored":136355840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53471584256,"internally_reserved":0,"allocated":81289216,"data_stored":81289216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:16.237 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:16.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:16.354+0000 7fe4750d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:16.364+0000 7fe4750d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:16.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:45:16.589 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-09-11T14:45:16.613 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-11T14:45:15.925373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":371960,"kb_used_data":371960,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156521224,"statfs":{"total":160658620416,"available":160277733376,"internally_reserved":0,"allocated":380887040,"data_stored":380887040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168035816,"num_objects":-228,"num_object_clones":0,"num_object_copies":-456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28555,"num_read_kb":-141621,"num_write":-1614,"num_write_kb":-205412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001019"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.075066+0000","last_change":"2024-09-11T14:35:14.942992+0000","last_active":"2024-09-11T14:45:13.075066+0000","last_peered":"2024-09-11T14:45:13.075066+0000","last_clean":"2024-09-11T14:45:13.075066+0000","last_became_active":"2024-09-11T14:35:12.958460+0000","last_became_peered":"2024-09-11T14:35:12.958460+0000","last_unstale":"2024-09-11T14:45:13.075066+0000","last_undegraded":"2024-09-11T14:45:13.075066+0000","last_fullsized":"2024-09-11T14:45:13.075066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.112372+0000","last_change":"2024-09-11T14:35:14.946690+0000","last_active":"2024-09-11T14:45:13.112372+0000","last_peered":"2024-09-11T14:45:13.112372+0000","last_clean":"2024-09-11T14:45:13.112372+0000","last_became_active":"2024-09-11T14:35:13.015169+0000","last_became_peered":"2024-09-11T14:35:13.015169+0000","last_unstale":"2024-09-11T14:45:13.112372+0000","last_undegraded":"2024-09-11T14:45:13.112372+0000","last_fullsized":"2024-09-11T14:45:13.112372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.082880+0000","last_change":"2024-09-11T14:35:14.942753+0000","last_active":"2024-09-11T14:45:13.082880+0000","last_peered":"2024-09-11T14:45:13.082880+0000","last_clean":"2024-09-11T14:45:13.082880+0000","last_became_active":"2024-09-11T14:35:12.989419+0000","last_became_peered":"2024-09-11T14:35:12.989419+0000","last_unstale":"2024-09-11T14:45:13.082880+0000","last_undegraded":"2024-09-11T14:45:13.082880+0000","last_fullsized":"2024-09-11T14:45:13.082880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.073328+0000","last_change":"2024-09-11T14:35:14.947434+0000","last_active":"2024-09-11T14:45:13.073328+0000","last_peered":"2024-09-11T14:45:13.073328+0000","last_clean":"2024-09-11T14:45:13.073328+0000","last_became_active":"2024-09-11T14:35:12.976308+0000","last_became_peered":"2024-09-11T14:35:12.976308+0000","last_unstale":"2024-09-11T14:45:13.073328+0000","last_undegraded":"2024-09-11T14:45:13.073328+0000","last_fullsized":"2024-09-11T14:45:13.073328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":323,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-11T14:45:11.126639+0000","last_change":"2024-09-11T14:35:14.951771+0000","last_active":"2024-09-11T14:45:11.126639+0000","last_peered":"2024-09-11T14:45:11.126639+0000","last_clean":"2024-09-11T14:45:11.126639+0000","last_became_active":"2024-09-11T14:35:12.981976+0000","last_became_peered":"2024-09-11T14:35:12.981976+0000","last_unstale":"2024-09-11T14:45:11.126639+0000","last_undegraded":"2024-09-11T14:45:11.126639+0000","last_fullsized":"2024-09-11T14:45:11.126639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.151237+0000","last_change":"2024-09-11T14:35:14.951268+0000","last_active":"2024-09-11T14:45:13.151237+0000","last_peered":"2024-09-11T14:45:13.151237+0000","last_clean":"2024-09-11T14:45:13.151237+0000","last_became_active":"2024-09-11T14:35:12.945557+0000","last_became_peered":"2024-09-11T14:35:12.945557+0000","last_unstale":"2024-09-11T14:45:13.151237+0000","last_undegraded":"2024-09-11T14:45:13.151237+0000","last_fullsized":"2024-09-11T14:45:13.151237+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.153611+0000","last_change":"2024-09-11T14:35:14.944204+0000","last_active":"2024-09-11T14:45:13.153611+0000","last_peered":"2024-09-11T14:45:13.153611+0000","last_clean":"2024-09-11T14:45:13.153611+0000","last_became_active":"2024-09-11T14:35:12.965045+0000","last_became_peered":"2024-09-11T14:35:12.965045+0000","last_unstale":"2024-09-11T14:45:13.153611+0000","last_undegraded":"2024-09-11T14:45:13.153611+0000","last_fullsized":"2024-09-11T14:45:13.153611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":328,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.089773+0000","last_change":"2024-09-11T14:35:14.942049+0000","last_active":"2024-09-11T14:45:13.089773+0000","last_peered":"2024-09-11T14:45:13.089773+0000","last_clean":"2024-09-11T14:45:13.089773+0000","last_became_active":"2024-09-11T14:35:12.944374+0000","last_became_peered":"2024-09-11T14:35:12.944374+0000","last_unstale":"2024-09-11T14:45:13.089773+0000","last_undegraded":"2024-09-11T14:45:13.089773+0000","last_fullsized":"2024-09-11T14:45:13.089773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":399,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.112179+0000","last_change":"2024-09-11T14:35:10.963766+0000","last_active":"2024-09-11T14:45:13.112179+0000","last_peered":"2024-09-11T14:45:13.112179+0000","last_clean":"2024-09-11T14:45:13.112179+0000","last_became_active":"2024-09-11T14:35:10.942660+0000","last_became_peered":"2024-09-11T14:35:10.942660+0000","last_unstale":"2024-09-11T14:45:13.112179+0000","last_undegraded":"2024-09-11T14:45:13.112179+0000","last_fullsized":"2024-09-11T14:45:13.112179+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_clean_scrub_stamp":"2024-09-11T14:35:09.830264+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160281124864,"internally_reserved":0,"allocated":377495552,"data_stored":377495552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":80684,"kb_used_data":80684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52217044,"statfs":{"total":53552873472,"available":53470253056,"internally_reserved":0,"allocated":82620416,"data_stored":82620416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":157332,"kb_used_data":157332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52140396,"statfs":{"total":53552873472,"available":53391765504,"internally_reserved":0,"allocated":161107968,"data_stored":161107968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":133944,"kb_used_data":133944,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52163784,"statfs":{"total":53552873472,"available":53415714816,"internally_reserved":0,"allocated":137158656,"data_stored":137158656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53416517632,"internally_reserved":0,"allocated":136355840,"data_stored":136355840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53471584256,"internally_reserved":0,"allocated":81289216,"data_stored":81289216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:16.614 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:45:16.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:16.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:16.732+0000 7f6a98f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:16.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:16.744+0000 7f6a98f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:16.963 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:45:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-09-11T14:45:16.987 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-11T14:45:15.925373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":371960,"kb_used_data":371960,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156521224,"statfs":{"total":160658620416,"available":160277733376,"internally_reserved":0,"allocated":380887040,"data_stored":380887040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168035816,"num_objects":-228,"num_object_clones":0,"num_object_copies":-456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28555,"num_read_kb":-141621,"num_write":-1614,"num_write_kb":-205412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-79,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001019"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.075066+0000","last_change":"2024-09-11T14:35:14.942992+0000","last_active":"2024-09-11T14:45:13.075066+0000","last_peered":"2024-09-11T14:45:13.075066+0000","last_clean":"2024-09-11T14:45:13.075066+0000","last_became_active":"2024-09-11T14:35:12.958460+0000","last_became_peered":"2024-09-11T14:35:12.958460+0000","last_unstale":"2024-09-11T14:45:13.075066+0000","last_undegraded":"2024-09-11T14:45:13.075066+0000","last_fullsized":"2024-09-11T14:45:13.075066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.112372+0000","last_change":"2024-09-11T14:35:14.946690+0000","last_active":"2024-09-11T14:45:13.112372+0000","last_peered":"2024-09-11T14:45:13.112372+0000","last_clean":"2024-09-11T14:45:13.112372+0000","last_became_active":"2024-09-11T14:35:13.015169+0000","last_became_peered":"2024-09-11T14:35:13.015169+0000","last_unstale":"2024-09-11T14:45:13.112372+0000","last_undegraded":"2024-09-11T14:45:13.112372+0000","last_fullsized":"2024-09-11T14:45:13.112372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.082880+0000","last_change":"2024-09-11T14:35:14.942753+0000","last_active":"2024-09-11T14:45:13.082880+0000","last_peered":"2024-09-11T14:45:13.082880+0000","last_clean":"2024-09-11T14:45:13.082880+0000","last_became_active":"2024-09-11T14:35:12.989419+0000","last_became_peered":"2024-09-11T14:35:12.989419+0000","last_unstale":"2024-09-11T14:45:13.082880+0000","last_undegraded":"2024-09-11T14:45:13.082880+0000","last_fullsized":"2024-09-11T14:45:13.082880+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.073328+0000","last_change":"2024-09-11T14:35:14.947434+0000","last_active":"2024-09-11T14:45:13.073328+0000","last_peered":"2024-09-11T14:45:13.073328+0000","last_clean":"2024-09-11T14:45:13.073328+0000","last_became_active":"2024-09-11T14:35:12.976308+0000","last_became_peered":"2024-09-11T14:35:12.976308+0000","last_unstale":"2024-09-11T14:45:13.073328+0000","last_undegraded":"2024-09-11T14:45:13.073328+0000","last_fullsized":"2024-09-11T14:45:13.073328+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":323,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-11T14:45:11.126639+0000","last_change":"2024-09-11T14:35:14.951771+0000","last_active":"2024-09-11T14:45:11.126639+0000","last_peered":"2024-09-11T14:45:11.126639+0000","last_clean":"2024-09-11T14:45:11.126639+0000","last_became_active":"2024-09-11T14:35:12.981976+0000","last_became_peered":"2024-09-11T14:35:12.981976+0000","last_unstale":"2024-09-11T14:45:11.126639+0000","last_undegraded":"2024-09-11T14:45:11.126639+0000","last_fullsized":"2024-09-11T14:45:11.126639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.151237+0000","last_change":"2024-09-11T14:35:14.951268+0000","last_active":"2024-09-11T14:45:13.151237+0000","last_peered":"2024-09-11T14:45:13.151237+0000","last_clean":"2024-09-11T14:45:13.151237+0000","last_became_active":"2024-09-11T14:35:12.945557+0000","last_became_peered":"2024-09-11T14:35:12.945557+0000","last_unstale":"2024-09-11T14:45:13.151237+0000","last_undegraded":"2024-09-11T14:45:13.151237+0000","last_fullsized":"2024-09-11T14:45:13.151237+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.153611+0000","last_change":"2024-09-11T14:35:14.944204+0000","last_active":"2024-09-11T14:45:13.153611+0000","last_peered":"2024-09-11T14:45:13.153611+0000","last_clean":"2024-09-11T14:45:13.153611+0000","last_became_active":"2024-09-11T14:35:12.965045+0000","last_became_peered":"2024-09-11T14:35:12.965045+0000","last_unstale":"2024-09-11T14:45:13.153611+0000","last_undegraded":"2024-09-11T14:45:13.153611+0000","last_fullsized":"2024-09-11T14:45:13.153611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":328,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.089773+0000","last_change":"2024-09-11T14:35:14.942049+0000","last_active":"2024-09-11T14:45:13.089773+0000","last_peered":"2024-09-11T14:45:13.089773+0000","last_clean":"2024-09-11T14:45:13.089773+0000","last_became_active":"2024-09-11T14:35:12.944374+0000","last_became_peered":"2024-09-11T14:35:12.944374+0000","last_unstale":"2024-09-11T14:45:13.089773+0000","last_undegraded":"2024-09-11T14:45:13.089773+0000","last_fullsized":"2024-09-11T14:45:13.089773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":399,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T14:45:13.112179+0000","last_change":"2024-09-11T14:35:10.963766+0000","last_active":"2024-09-11T14:45:13.112179+0000","last_peered":"2024-09-11T14:45:13.112179+0000","last_clean":"2024-09-11T14:45:13.112179+0000","last_became_active":"2024-09-11T14:35:10.942660+0000","last_became_peered":"2024-09-11T14:35:10.942660+0000","last_unstale":"2024-09-11T14:45:13.112179+0000","last_undegraded":"2024-09-11T14:45:13.112179+0000","last_fullsized":"2024-09-11T14:45:13.112179+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_clean_scrub_stamp":"2024-09-11T14:35:09.830264+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160281124864,"internally_reserved":0,"allocated":377495552,"data_stored":377495552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":80684,"kb_used_data":80684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52217044,"statfs":{"total":53552873472,"available":53470253056,"internally_reserved":0,"allocated":82620416,"data_stored":82620416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":157332,"kb_used_data":157332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52140396,"statfs":{"total":53552873472,"available":53391765504,"internally_reserved":0,"allocated":161107968,"data_stored":161107968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":133944,"kb_used_data":133944,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52163784,"statfs":{"total":53552873472,"available":53415714816,"internally_reserved":0,"allocated":137158656,"data_stored":137158656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53416517632,"internally_reserved":0,"allocated":136355840,"data_stored":136355840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53393022976,"internally_reserved":0,"allocated":159850496,"data_stored":159850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53471584256,"internally_reserved":0,"allocated":81289216,"data_stored":81289216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:16.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:45:17.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:17.168+0000 7fb3840bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:17.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:17.181+0000 7fb3840bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:17.582 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:45:17.582 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":121,"fsid":"b33ea6be-da0e-4fe6-a2f2-8f9cbab56580","created":"2024-09-11T14:35:00.967980+0000","modified":"2024-09-11T14:45:15.010980+0000","last_up_change":"2024-09-11T14:35:08.819126+0000","last_in_change":"2024-09-11T14:35:02.879246+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:35:09.770562+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:35:10.863372+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9fb327d3-b49a-4c74-be15-20ef7244a835","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1512393070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":1512393070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1512393070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1512393070}]},"public_addr":"172.21.15.40:6808/1512393070","cluster_addr":"172.21.15.40:6809/1512393070","heartbeat_back_addr":"172.21.15.40:6806/1512393070","heartbeat_front_addr":"172.21.15.40:6818/1512393070","state":["exists","up"]},{"osd":1,"uuid":"0ba9f622-b79e-472e-8c04-96b768f7bc6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":991486483}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":991486483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":991486483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":991486483}]},"public_addr":"172.21.15.40:6803/991486483","cluster_addr":"172.21.15.40:6804/991486483","heartbeat_back_addr":"172.21.15.40:6802/991486483","heartbeat_front_addr":"172.21.15.40:6805/991486483","state":["exists","up"]},{"osd":2,"uuid":"abd80b24-fbd4-4e2e-bdb6-8b8052d07ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3190718975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":3190718975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3190718975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6817","nonce":3190718975}]},"public_addr":"172.21.15.40:6810/3190718975","cluster_addr":"172.21.15.40:6807/3190718975","heartbeat_back_addr":"172.21.15.40:6816/3190718975","heartbeat_front_addr":"172.21.15.40:6817/3190718975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":13},{"begin":17,"length":1},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":28,"length":8}]},{"pool":4,"snaps":[{"begin":2,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:45:18.604 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T14:45:18.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:45:18.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:18.722+0000 7fe789ad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:18.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:18.737+0000 7fe789ad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:18.820 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-11T14:45:18.821 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "" 2024-09-11T14:45:18.821 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-11T14:45:18.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T14:45:18.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:18.956+0000 7fd485d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:18.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:18.967+0000 7fd485d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:19.172 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T14:45:19.188 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T14:45:19.188 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:45:19.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:19.304+0000 7f10138fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:19.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:19.319+0000 7f10138fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:19.341 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-11T14:45:19.341 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "" 2024-09-11T14:45:19.341 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-11T14:45:19.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T14:45:19.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:19.473+0000 7fcbdcf11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:19.483+0000 7fcbdcf11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T14:45:19.897 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T14:45:19.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:45:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:20.016+0000 7f22a00bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:20.029+0000 7f22a00bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.044 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-11T14:45:20.044 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "" 2024-09-11T14:45:20.045 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-11T14:45:20.056 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T14:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:20.163+0000 7f487e4a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:20.172+0000 7f487e4a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T14:45:20.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:20.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:20.494+0000 7fb739b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:20.502+0000 7fb739b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:20.695 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-09-11T14:45:20.708 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-11T14:45:19.926212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":199712,"kb_used_data":199712,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156693472,"statfs":{"total":160658620416,"available":160454115328,"internally_reserved":0,"allocated":204505088,"data_stored":204505088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168035623,"num_objects":-220,"num_object_clones":0,"num_object_copies":-440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28135,"num_read_kb":-141459,"num_write":-1515,"num_write_kb":-205315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.913049"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:15.047956+0000","last_change":"2024-09-11T14:35:14.942992+0000","last_active":"2024-09-11T14:45:15.047956+0000","last_peered":"2024-09-11T14:45:15.047956+0000","last_clean":"2024-09-11T14:45:15.047956+0000","last_became_active":"2024-09-11T14:35:12.958460+0000","last_became_peered":"2024-09-11T14:35:12.958460+0000","last_unstale":"2024-09-11T14:45:15.047956+0000","last_undegraded":"2024-09-11T14:45:15.047956+0000","last_fullsized":"2024-09-11T14:45:15.047956+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":333,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:17.152531+0000","last_change":"2024-09-11T14:35:14.946690+0000","last_active":"2024-09-11T14:45:17.152531+0000","last_peered":"2024-09-11T14:45:17.152531+0000","last_clean":"2024-09-11T14:45:17.152531+0000","last_became_active":"2024-09-11T14:35:13.015169+0000","last_became_peered":"2024-09-11T14:35:13.015169+0000","last_unstale":"2024-09-11T14:45:17.152531+0000","last_undegraded":"2024-09-11T14:45:17.152531+0000","last_fullsized":"2024-09-11T14:45:17.152531+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":333,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:15.070566+0000","last_change":"2024-09-11T14:35:14.942753+0000","last_active":"2024-09-11T14:45:15.070566+0000","last_peered":"2024-09-11T14:45:15.070566+0000","last_clean":"2024-09-11T14:45:15.070566+0000","last_became_active":"2024-09-11T14:35:12.989419+0000","last_became_peered":"2024-09-11T14:35:12.989419+0000","last_unstale":"2024-09-11T14:45:15.070566+0000","last_undegraded":"2024-09-11T14:45:15.070566+0000","last_fullsized":"2024-09-11T14:45:15.070566+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":333,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:17.238501+0000","last_change":"2024-09-11T14:35:14.947434+0000","last_active":"2024-09-11T14:45:17.238501+0000","last_peered":"2024-09-11T14:45:17.238501+0000","last_clean":"2024-09-11T14:45:17.238501+0000","last_became_active":"2024-09-11T14:35:12.976308+0000","last_became_peered":"2024-09-11T14:35:12.976308+0000","last_unstale":"2024-09-11T14:45:17.238501+0000","last_undegraded":"2024-09-11T14:45:17.238501+0000","last_fullsized":"2024-09-11T14:45:17.238501+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":335,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:15.135908+0000","last_change":"2024-09-11T14:35:14.951771+0000","last_active":"2024-09-11T14:45:15.135908+0000","last_peered":"2024-09-11T14:45:15.135908+0000","last_clean":"2024-09-11T14:45:15.135908+0000","last_became_active":"2024-09-11T14:35:12.981976+0000","last_became_peered":"2024-09-11T14:35:12.981976+0000","last_unstale":"2024-09-11T14:45:15.135908+0000","last_undegraded":"2024-09-11T14:45:15.135908+0000","last_fullsized":"2024-09-11T14:45:15.135908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":333,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:15.102660+0000","last_change":"2024-09-11T14:35:14.951268+0000","last_active":"2024-09-11T14:45:15.102660+0000","last_peered":"2024-09-11T14:45:15.102660+0000","last_clean":"2024-09-11T14:45:15.102660+0000","last_became_active":"2024-09-11T14:35:12.945557+0000","last_became_peered":"2024-09-11T14:35:12.945557+0000","last_unstale":"2024-09-11T14:45:15.102660+0000","last_undegraded":"2024-09-11T14:45:15.102660+0000","last_fullsized":"2024-09-11T14:45:15.102660+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":333,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:15.100343+0000","last_change":"2024-09-11T14:35:14.944204+0000","last_active":"2024-09-11T14:45:15.100343+0000","last_peered":"2024-09-11T14:45:15.100343+0000","last_clean":"2024-09-11T14:45:15.100343+0000","last_became_active":"2024-09-11T14:35:12.965045+0000","last_became_peered":"2024-09-11T14:35:12.965045+0000","last_unstale":"2024-09-11T14:45:15.100343+0000","last_undegraded":"2024-09-11T14:45:15.100343+0000","last_fullsized":"2024-09-11T14:45:15.100343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":334,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:15.067611+0000","last_change":"2024-09-11T14:35:14.942049+0000","last_active":"2024-09-11T14:45:15.067611+0000","last_peered":"2024-09-11T14:45:15.067611+0000","last_clean":"2024-09-11T14:45:15.067611+0000","last_became_active":"2024-09-11T14:35:12.944374+0000","last_became_peered":"2024-09-11T14:35:12.944374+0000","last_unstale":"2024-09-11T14:45:15.067611+0000","last_undegraded":"2024-09-11T14:45:15.067611+0000","last_fullsized":"2024-09-11T14:45:15.067611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:11.839082+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":405,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:15.098964+0000","last_change":"2024-09-11T14:35:10.963766+0000","last_active":"2024-09-11T14:45:15.098964+0000","last_peered":"2024-09-11T14:45:15.098964+0000","last_clean":"2024-09-11T14:45:15.098964+0000","last_became_active":"2024-09-11T14:35:10.942660+0000","last_became_peered":"2024-09-11T14:35:10.942660+0000","last_unstale":"2024-09-11T14:45:15.098964+0000","last_undegraded":"2024-09-11T14:45:15.098964+0000","last_fullsized":"2024-09-11T14:45:15.098964+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:35:09.830264+0000","last_clean_scrub_stamp":"2024-09-11T14:35:09.830264+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160525561856,"internally_reserved":0,"allocated":133058560,"data_stored":133058560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53521022976,"internally_reserved":0,"allocated":31850496,"data_stored":31850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706272,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":19872,"kb_used_data":19872,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52277856,"statfs":{"total":53552873472,"available":53532524544,"internally_reserved":0,"allocated":20348928,"data_stored":20348928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":63676,"kb_used_data":63676,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52234052,"statfs":{"total":53552873472,"available":53487669248,"internally_reserved":0,"allocated":65204224,"data_stored":65204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":116164,"kb_used_data":116164,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52181564,"statfs":{"total":53552873472,"available":53433921536,"internally_reserved":0,"allocated":118951936,"data_stored":118951936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53521022976,"internally_reserved":0,"allocated":31850496,"data_stored":31850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53452849152,"internally_reserved":0,"allocated":100024320,"data_stored":100024320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53521022976,"internally_reserved":0,"allocated":31850496,"data_stored":31850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551689728,"internally_reserved":0,"allocated":1183744,"data_stored":1183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:20.711 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T14:35:11.839082+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.711 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T14:35:11.839082+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.711 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T14:35:11.839082+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.711 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T14:35:11.839082+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.712 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T14:35:11.839082+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.712 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T14:35:11.839082+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.712 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T14:35:11.839082+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.712 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T14:35:11.839082+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.712 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T14:35:09.830264+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=35, tm_sec=9, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=45, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:45:20.712 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T14:45:40.713 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:45:40.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:40.836+0000 7facd4b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:40.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:40.844+0000 7facd4b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:41.039 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-11T14:45:41.040 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-09-11T14:45:41.052 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-11T14:45:39.931800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5548,"kb_used_data":5548,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156887636,"statfs":{"total":160658620416,"available":160652939264,"internally_reserved":0,"allocated":5681152,"data_stored":5681152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003301"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":335,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.883547+0000","last_change":"2024-09-11T14:45:19.938338+0000","last_active":"2024-09-11T14:45:19.883547+0000","last_peered":"2024-09-11T14:45:19.883547+0000","last_clean":"2024-09-11T14:45:19.883547+0000","last_became_active":"2024-09-11T14:35:12.958460+0000","last_became_peered":"2024-09-11T14:35:12.958460+0000","last_unstale":"2024-09-11T14:45:19.883547+0000","last_undegraded":"2024-09-11T14:45:19.883547+0000","last_fullsized":"2024-09-11T14:45:19.883547+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:45:19.938156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:19.938156+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":335,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.883392+0000","last_change":"2024-09-11T14:45:19.930795+0000","last_active":"2024-09-11T14:45:19.883392+0000","last_peered":"2024-09-11T14:45:19.883392+0000","last_clean":"2024-09-11T14:45:19.883392+0000","last_became_active":"2024-09-11T14:35:13.015169+0000","last_became_peered":"2024-09-11T14:35:13.015169+0000","last_unstale":"2024-09-11T14:45:19.883392+0000","last_undegraded":"2024-09-11T14:45:19.883392+0000","last_fullsized":"2024-09-11T14:45:19.883392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:45:19.930604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:19.930604+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":335,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.880591+0000","last_change":"2024-09-11T14:45:19.935104+0000","last_active":"2024-09-11T14:45:19.880591+0000","last_peered":"2024-09-11T14:45:19.880591+0000","last_clean":"2024-09-11T14:45:19.880591+0000","last_became_active":"2024-09-11T14:35:12.989419+0000","last_became_peered":"2024-09-11T14:35:12.989419+0000","last_unstale":"2024-09-11T14:45:19.880591+0000","last_undegraded":"2024-09-11T14:45:19.880591+0000","last_fullsized":"2024-09-11T14:45:19.880591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:45:19.934912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:19.934912+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":335,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.879504+0000","last_change":"2024-09-11T14:45:19.909838+0000","last_active":"2024-09-11T14:45:19.879504+0000","last_peered":"2024-09-11T14:45:19.879504+0000","last_clean":"2024-09-11T14:45:19.879504+0000","last_became_active":"2024-09-11T14:35:12.976308+0000","last_became_peered":"2024-09-11T14:35:12.976308+0000","last_unstale":"2024-09-11T14:45:19.879504+0000","last_undegraded":"2024-09-11T14:45:19.879504+0000","last_fullsized":"2024-09-11T14:45:19.879504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:45:19.909656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:19.909656+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":337,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.173500+0000","last_change":"2024-09-11T14:45:19.229014+0000","last_active":"2024-09-11T14:45:19.173500+0000","last_peered":"2024-09-11T14:45:19.173500+0000","last_clean":"2024-09-11T14:45:19.173500+0000","last_became_active":"2024-09-11T14:35:12.981976+0000","last_became_peered":"2024-09-11T14:35:12.981976+0000","last_unstale":"2024-09-11T14:45:19.173500+0000","last_undegraded":"2024-09-11T14:45:19.173500+0000","last_fullsized":"2024-09-11T14:45:19.173500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2024-09-11T14:45:19.228803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:19.228803+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":335,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.372828+0000","last_change":"2024-09-11T14:45:20.387317+0000","last_active":"2024-09-11T14:45:20.372828+0000","last_peered":"2024-09-11T14:45:20.372828+0000","last_clean":"2024-09-11T14:45:20.372828+0000","last_became_active":"2024-09-11T14:35:12.945557+0000","last_became_peered":"2024-09-11T14:35:12.945557+0000","last_unstale":"2024-09-11T14:45:20.372828+0000","last_undegraded":"2024-09-11T14:45:20.372828+0000","last_fullsized":"2024-09-11T14:45:20.372828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:45:20.387067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:20.387067+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":335,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:20.371019+0000","last_change":"2024-09-11T14:45:20.385562+0000","last_active":"2024-09-11T14:45:20.371019+0000","last_peered":"2024-09-11T14:45:20.371019+0000","last_clean":"2024-09-11T14:45:20.371019+0000","last_became_active":"2024-09-11T14:35:12.965045+0000","last_became_peered":"2024-09-11T14:35:12.965045+0000","last_unstale":"2024-09-11T14:45:20.371019+0000","last_undegraded":"2024-09-11T14:45:20.371019+0000","last_fullsized":"2024-09-11T14:45:20.371019+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:45:20.385346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:20.385346+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":336,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.879090+0000","last_change":"2024-09-11T14:45:19.942662+0000","last_active":"2024-09-11T14:45:19.879090+0000","last_peered":"2024-09-11T14:45:19.879090+0000","last_clean":"2024-09-11T14:45:19.879090+0000","last_became_active":"2024-09-11T14:35:12.944374+0000","last_became_peered":"2024-09-11T14:35:12.944374+0000","last_unstale":"2024-09-11T14:45:19.879090+0000","last_undegraded":"2024-09-11T14:45:19.879090+0000","last_fullsized":"2024-09-11T14:45:19.879090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'1","last_scrub_stamp":"2024-09-11T14:45:19.942457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:19.942457+0000","last_clean_scrub_stamp":"2024-09-11T14:35:11.839082+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":407,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-11T14:45:19.877225+0000","last_change":"2024-09-11T14:45:19.968294+0000","last_active":"2024-09-11T14:45:19.877225+0000","last_peered":"2024-09-11T14:45:19.877225+0000","last_clean":"2024-09-11T14:45:19.877225+0000","last_became_active":"2024-09-11T14:35:10.942660+0000","last_became_peered":"2024-09-11T14:35:10.942660+0000","last_unstale":"2024-09-11T14:45:19.877225+0000","last_undegraded":"2024-09-11T14:45:19.877225+0000","last_fullsized":"2024-09-11T14:45:19.877225+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-09-11T14:45:19.968072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:45:19.968072+0000","last_clean_scrub_stamp":"2024-09-11T14:35:09.830264+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160652939264,"internally_reserved":0,"allocated":5681152,"data_stored":5681152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706292,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1156,"kb_used_data":1156,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296572,"statfs":{"total":53552873472,"available":53551689728,"internally_reserved":0,"allocated":1183744,"data_stored":1183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2332,"kb_used_data":2332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295396,"statfs":{"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706283,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2060,"kb_used_data":2060,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295668,"statfs":{"total":53552873472,"available":53550764032,"internally_reserved":0,"allocated":2109440,"data_stored":2109440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550764032,"internally_reserved":0,"allocated":2109440,"data_stored":2109440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550485504,"internally_reserved":0,"allocated":2387968,"data_stored":2387968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551689728,"internally_reserved":0,"allocated":1183744,"data_stored":1183744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:45:41.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T14:45:41.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:41.158+0000 7f5223391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:41.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-11T14:45:41.170+0000 7f5223391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:45:41.392 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T14:45:41.393 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T14:45:41.393 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T14:45:41.393 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:45:41.473 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T14:45:41.473 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T14:45:41.473 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:45:41.553 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T14:45:41.553 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T14:45:41.553 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:45:41.633 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T14:45:41.634 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T14:45:41.634 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T14:45:41.634 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:45:41.664 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T14:45:41.664 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T14:45:41.665 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T14:45:41.665 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:45:41.717 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T14:45:41.717 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T14:45:41.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(POOL_FULL\)' | head -n 1 2024-09-11T14:45:41.791 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:45:41.791 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T14:45:41.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:45:41.965 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T14:45:42.029 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:45:42.029 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T14:45:42.079 INFO:tasks.ceph:Archiving mon data... 2024-09-11T14:45:42.080 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901361/data/mon.a.tgz 2024-09-11T14:45:42.080 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-09-11T14:45:42.099 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.ofUMCOtsuQ 2024-09-11T14:45:42.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ofUMCOtsuQ 2024-09-11T14:45:42.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.ofUMCOtsuQ 2024-09-11T14:45:42.311 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.ofUMCOtsuQ is 227KB 2024-09-11T14:45:42.405 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.ofUMCOtsuQ 2024-09-11T14:45:42.420 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T14:45:42.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T14:45:42.542 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T14:45:42.542 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T14:45:42.544 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901361/remote/smithi040/crash 2024-09-11T14:45:42.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T14:45:42.607 INFO:tasks.ceph:Compressing logs... 2024-09-11T14:45:42.607 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T14:45:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T14:45:42.672 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29271.log 2024-09-11T14:45:42.692 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29397.log 2024-09-11T14:45:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.29271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29313.log 2024-09-11T14:45:42.693 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29271.log.gz 2024-09-11T14:45:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29120.log 2024-09-11T14:45:42.693 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29397.log.gz 2024-09-11T14:45:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32036.log 2024-09-11T14:45:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29313.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29313.log.gz 2024-09-11T14:45:42.693 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32078.log 2024-09-11T14:45:42.694 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29120.log.gz 2024-09-11T14:45:42.694 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.32036.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29565.log 2024-09-11T14:45:42.694 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32036.log.gz 2024-09-11T14:45:42.694 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31910.log 2024-09-11T14:45:42.694 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32078.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32078.log.gz 2024-09-11T14:45:42.694 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29565.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T14:45:42.694 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29565.log.gz 2024-09-11T14:45:42.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29478.log 2024-09-11T14:45:42.695 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31910.log.gz 2024-09-11T14:45:42.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27553.log 2024-09-11T14:45:42.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29604.log 2024-09-11T14:45:42.695 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29478.log.gz 2024-09-11T14:45:42.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.tmp-client.admin.27553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28503.log 2024-09-11T14:45:42.695 INFO:teuthology.orchestra.run.smithi040.stderr: 66.4% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27553.log.gz 2024-09-11T14:45:42.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29355.log 2024-09-11T14:45:42.696 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29604.log.gz 2024-09-11T14:45:42.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32120.log 2024-09-11T14:45:42.696 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28503.log.gz 2024-09-11T14:45:42.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29355.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T14:45:42.696 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29355.log.gz 2024-09-11T14:45:42.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32120.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32120.log.gz 2024-09-11T14:45:42.696 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29817.log 2024-09-11T14:45:42.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32162.log 2024-09-11T14:45:42.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29817.log: gzip 87.0% -5 -- replaced with /var/log/ceph/ceph.audit.log.gz --verbose 2024-09-11T14:45:42.697 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.32204.log 2024-09-11T14:45:42.697 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29817.log.gz 2024-09-11T14:45:42.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32244.log 2024-09-11T14:45:42.697 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32162.log.gz 2024-09-11T14:45:42.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28592.log 2024-09-11T14:45:42.697 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32204.log.gz 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31868.log 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32244.log.gz 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28702.log 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28592.log.gz 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28951.log 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31868.log.gz 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28702.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28702.log.gz 2024-09-11T14:45:42.698 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28951.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28951.log.gz 2024-09-11T14:45:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31994.log 2024-09-11T14:45:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31784.log 2024-09-11T14:45:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29711.log 2024-09-11T14:45:42.699 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31994.log.gz 2024-09-11T14:45:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28981.log 2024-09-11T14:45:42.699 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31784.log.gz 2024-09-11T14:45:42.699 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30119.log 2024-09-11T14:45:42.700 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29711.log.gz 2024-09-11T14:45:42.700 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29220.log 2024-09-11T14:45:42.700 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28981.log.gz 2024-09-11T14:45:42.700 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30119.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T14:45:42.700 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31826.log 2024-09-11T14:45:42.700 INFO:teuthology.orchestra.run.smithi040.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30119.log.gz 2024-09-11T14:45:42.700 INFO:teuthology.orchestra.run.smithi040.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.29220.log.gz 2024-09-11T14:45:42.700 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28742.log 2024-09-11T14:45:42.701 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31826.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T14:45:42.701 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31826.log.gz 2024-09-11T14:45:42.701 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28742.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T14:45:42.701 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28742.log.gz 2024-09-11T14:45:42.701 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29479.log 2024-09-11T14:45:42.701 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29477.log 2024-09-11T14:45:42.701 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29691.log 2024-09-11T14:45:42.701 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29479.log.gz 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29477.log: gzip -5 --verbose 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.29605.log 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29691.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29477.log.gz 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29691.log.gz 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31952.log 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29605.log: /var/log/ceph/ceph-client.admin.29176.log 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29605.log.gz/var/log/ceph/ceph-client.admin.31952.log: 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31952.log.gz 2024-09-11T14:45:42.702 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29731.log 2024-09-11T14:45:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29176.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29176.log.gz 2024-09-11T14:45:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29901.log 2024-09-11T14:45:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29731.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29731.log.gz 2024-09-11T14:45:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31742.log 2024-09-11T14:45:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29901.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29901.log.gz 2024-09-11T14:45:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28792.log 2024-09-11T14:45:42.703 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31742.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29859.log 2024-09-11T14:45:42.703 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31742.log.gz 2024-09-11T14:45:42.704 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28792.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28792.log.gz 2024-09-11T14:45:42.704 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29859.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29859.log.gz 2024-09-11T14:45:42.762 INFO:teuthology.orchestra.run.smithi040.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T14:45:42.906 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T14:45:46.592 INFO:teuthology.orchestra.run.smithi040.stderr: 95.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T14:45:47.022 INFO:teuthology.orchestra.run.smithi040.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T14:45:47.117 INFO:teuthology.orchestra.run.smithi040.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T14:45:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-11T14:45:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m4.457s 2024-09-11T14:45:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m12.472s 2024-09-11T14:45:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m0.567s 2024-09-11T14:45:47.120 INFO:tasks.ceph:Archiving logs... 2024-09-11T14:45:47.120 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901361/remote/smithi040/log 2024-09-11T14:45:47.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T14:45:48.879 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T14:45:48.888 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T14:45:48.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T14:45:48.959 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T14:45:48.967 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T14:45:48.968 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:45:48.990 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:45:48.995 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:45:48.995 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-09-11T14:45:48.995 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +852us[ +824us] +/- 52ms 2024-09-11T14:45:48.996 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 -577us[ -577us] +/- 52ms 2024-09-11T14:45:48.996 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 -524us[ -524us] +/- 83ms 2024-09-11T14:45:48.996 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:45:48.997 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T14:45:49.006 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T14:45:49.006 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T14:45:49.064 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T14:45:49.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:45:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:45:49.161 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T14:45:49.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T14:45:49.180 INFO:teuthology.task.internal:Duration was 1076.750446 seconds 2024-09-11T14:45:49.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T14:45:49.189 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T14:45:49.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T14:45:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:45:49.549 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T14:45:49.549 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-09-11T14:45:49.549 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T14:45:49.611 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T14:45:49.612 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T14:45:49.679 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T14:45:49.679 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T14:45:49.821 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T14:45:49.831 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T14:45:49.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T14:45:49.857 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T14:45:49.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:45:49.921 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-09-11T14:45:49.945 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:45:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:45:50.001 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T14:45:50.011 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T14:45:50.011 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901361/remote/smithi040 2024-09-11T14:45:50.012 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T14:45:50.075 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T14:45:50.076 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T14:45:50.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T14:45:50.136 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T14:45:50.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T14:45:50.150 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T14:45:50.150 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T14:45:50.181 INFO:teuthology.orchestra.run.smithi040.stdout: 265622 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 11 14:45 /home/ubuntu/cephtest 2024-09-11T14:45:50.182 INFO:teuthology.orchestra.run.smithi040.stdout: 406098 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 11 14:35 /home/ubuntu/cephtest/mnt.0 2024-09-11T14:45:50.182 INFO:teuthology.orchestra.run.smithi040.stdout: 406099 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 11 14:36 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T14:45:50.182 INFO:teuthology.orchestra.run.smithi040.stdout: 420949 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 14:36 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T14:45:50.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:45:50.183 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-11T14:45:50.183 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-11T14:45:50.190 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T14:45:50.199 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T14:45:50.209 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi040 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-11T14:45:50.213 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format} duration: 1076.7504456043243 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi040 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=34256468dda440d391fe1389507990e0 status: fail success: false 2024-09-11T14:45:50.213 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:45:50.295 INFO:teuthology.run:FAIL