2024-09-11T14:28:29.113 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T14:28:29.130 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:28:29.224 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901370 branch: main description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests} email: null first_in_suite: false job_id: '7901370' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(POOL_FULL\) sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a workunit: branch: wip-matanb-crimson-seastore-logs env: CRIMSON_COMPAT: '1' sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5813 sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-seastore-logs suite_path: /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa suite_relpath: qa suite_repo: https://github.com/matan-b/ceph.git suite_sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHrT/poIiaUu1xUmApBoEuHf1hr2WTVqxSj2beq/agv8s9W41ud2Wwvw868ZO18APM9N7E52uFZXRg/X3QVT6Uo= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh env: RBD_FEATURES: '61' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-11_13:56:22 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T14:28:29.224 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa; will attempt to use it 2024-09-11T14:28:29.225 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks 2024-09-11T14:28:29.225 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T14:28:29.227 INFO:teuthology.task.internal:Checking packages... 2024-09-11T14:28:29.249 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '0d61375b8ab53be4c233f5b93e463edfdff9316a' 2024-09-11T14:28:29.249 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T14:28:29.249 INFO:teuthology.packaging:ref: None 2024-09-11T14:28:29.249 INFO:teuthology.packaging:tag: None 2024-09-11T14:28:29.249 INFO:teuthology.packaging:branch: main 2024-09-11T14:28:29.250 INFO:teuthology.packaging:sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:28:29.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-11T14:28:29.453 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4872.g0d61375b 2024-09-11T14:28:29.455 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T14:28:29.462 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T14:28:29.462 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T14:28:29.467 INFO:teuthology.task.internal:Saving configuration 2024-09-11T14:28:29.480 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T14:28:29.489 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T14:28:29.517 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901370', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 14:23:43.667928', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHrT/poIiaUu1xUmApBoEuHf1hr2WTVqxSj2beq/agv8s9W41ud2Wwvw868ZO18APM9N7E52uFZXRg/X3QVT6Uo='} 2024-09-11T14:28:29.519 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T14:28:29.525 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-11T14:28:29.525 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T14:28:29.557 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe37afd3ac0>, signals=[15]) 2024-09-11T14:28:29.557 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T14:28:29.562 INFO:teuthology.task.internal:Opening connections... 2024-09-11T14:28:29.563 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T14:28:29.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:28:29.648 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T14:28:29.654 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-09-11T14:28:29.734 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-09-11T14:28:29.734 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-09-11T14:28:29.791 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T14:28:29.792 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T14:28:29.793 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-09-11T14:28:29.816 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T14:28:29.824 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T14:28:29.833 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T14:28:29.834 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-09-11T14:28:29.851 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T14:28:29.858 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T14:28:29.858 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-09-11T14:28:29.915 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T14:28:29.948 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T14:28:29.962 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-09-11T14:28:29.963 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-09-11T14:28:29.963 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T14:28:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:28:29.980 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-09-11T14:28:30.037 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-503.el9.x86_64 2024-09-11T14:28:30.038 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-503.el9.x86_64 2024-09-11T14:28:30.038 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-09-11T14:28:32.591 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-11T14:28:34.575 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-11T14:28:39.218 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-11T14:28:40.702 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 19 kB 00:00 2024-09-11T14:28:41.581 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-11T14:28:47.187 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-11T14:28:49.285 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T14:28:49.285 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T14:28:49.374 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:28:49.384 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:28:49.384 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-09-11T14:28:49.384 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:28:49.384 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:28:49.384 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T14:28:49.384 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:28:49.385 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T14:28:49.385 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T14:28:49.385 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T14:28:49.385 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:28:49.385 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:28:49.385 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:28:49.385 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-09-11T14:28:49.385 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:28:49.386 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 79 M 2024-09-11T14:28:49.386 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 126 M 2024-09-11T14:28:49.386 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:28:49.730 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 259 kB/s | 23 kB 00:00 2024-09-11T14:28:52.764 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.0 MB/s | 16 MB 00:03 2024-09-11T14:28:55.538 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 5.9 MB/s | 35 MB 00:05 2024-09-11T14:28:56.855 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 4.0 MB/s | 29 MB 00:07 2024-09-11T14:28:56.855 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:28:56.855 INFO:teuthology.orchestra.run.smithi059.stdout:Total 11 MB/s | 79 MB 00:07 2024-09-11T14:28:57.571 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:28:57.943 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:28:57.943 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:28:58.919 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:28:58.920 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:29:01.504 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:29:02.175 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:29:02.255 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:29:05.048 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:29:05.175 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:29:10.292 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:29:10.334 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:29:15.479 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:29:54.647 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:29:54.689 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:29:55.060 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:29:55.060 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:29:55.060 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:29:55.061 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:29:55.975 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:29:55.975 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:29:55.975 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:29:55.975 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T14:29:55.975 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T14:29:55.975 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T14:29:55.976 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T14:29:55.976 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:29:55.976 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:29:56.311 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:29:56.394 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:29:56.394 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:29:56.395 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T14:29:56.395 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T14:29:56.395 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T14:29:56.395 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T14:29:56.395 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T14:29:56.395 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:29:56.453 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:29:56.454 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:29:56.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 82.el9 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8274670 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-09-11T14:29:56.492 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-09-11T14:29:56.493 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:29:56.494 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T14:29:56.494 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T14:29:56.495 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-505.el9.x86_64 2024-09-11T14:29:56.495 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T14:29:57.082 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-09-11T14:29:58.377 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T14:29:58.413 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-09-11T14:29:58.415 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-09-11T14:29:58.430 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.pNag9B5Yx3 2024-09-11T14:29:58.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pNag9B5Yx3 2024-09-11T14:29:58.496 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.pNag9B5Yx3 2024-09-11T14:29:58.641 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.pNag9B5Yx3 is 6KB 2024-09-11T14:29:58.690 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.pNag9B5Yx3 2024-09-11T14:29:58.705 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T14:29:58.767 INFO:teuthology.orchestra.run.smithi059.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T14:29:58.768 INFO:teuthology.orchestra.run.smithi059.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T14:29:58.768 INFO:teuthology.orchestra.run.smithi059.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T14:29:58.768 INFO:teuthology.orchestra.run.smithi059.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T14:29:58.769 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T14:29:58.935 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-09-11T14:30:28.965 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T14:30:28.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-11T14:30:28.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:30:47.357 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2024-09-11T14:30:56.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-11T14:30:56.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:31:56.426 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T14:32:08.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-11T14:32:08.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:32:08.820 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-09-11T14:32:09.181 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-11T14:32:09.181 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T14:32:09.181 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-09-11T14:32:09.198 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-505.el9.x86_64 2024-09-11T14:32:09.198 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T14:32:09.199 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T14:32:09.199 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T14:32:10.200 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T14:32:10.200 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T14:32:10.275 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-09-11T14:32:10.296 DEBUG:teuthology.parallel:result is None 2024-09-11T14:32:10.296 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T14:32:10.303 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T14:32:10.303 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T14:32:10.321 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T14:32:10.328 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T14:32:10.335 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T14:32:10.336 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T14:32:10.390 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T14:32:10.396 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T14:32:10.397 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T14:32:10.466 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:32:10.478 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:32:10.479 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T14:32:10.486 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T14:32:10.486 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T14:32:10.550 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T14:32:10.559 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T14:32:10.559 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T14:32:10.608 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:32:10.695 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:32:10.745 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:32:10.830 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:32:10.873 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:32:10.874 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T14:32:10.942 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-09-11T14:32:11.028 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:32:11.162 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T14:32:11.169 INFO:teuthology.task.internal:Starting timer... 2024-09-11T14:32:11.169 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T14:32:11.195 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T14:32:11.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-09-11T14:32:11.289 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-09-11T14:32:11.291 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T14:32:11.291 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-09-11T14:32:11.321 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-09-11T14:32:11.322 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive'} 2024-09-11T14:32:11.322 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:32:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:32:11.390 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T14:32:11.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-09-11T14:32:11.465 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T14:32:11.474 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T14:32:11.487 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T14:32:11.488 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T14:35:55.240 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com')] 2024-09-11T14:35:55.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-11T14:35:55.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:35:55.321 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-09-11T14:35:55.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-09-11T14:35:55.401 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T14:35:55.417 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T14:35:55.418 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T14:35:55.418 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:35:55.476 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T14:35:55.491 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T14:35:55.518 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-09-11T14:35:55.530 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-09-11T14:35:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T14:35:55.560 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T14:35:55.622 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:35:55.627 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:35:55.627 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-09-11T14:35:55.627 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:35:55.627 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:35:55.627 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:35:55.627 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:35:55.628 INFO:teuthology.run_tasks:Running task install... 2024-09-11T14:35:55.636 DEBUG:teuthology.task.install:project ceph 2024-09-11T14:35:55.636 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}} 2024-09-11T14:35:55.636 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'} 2024-09-11T14:35:55.636 INFO:teuthology.task.install:Using flavor: crimson 2024-09-11T14:35:55.641 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T14:35:55.641 INFO:teuthology.task.install:extra packages: [] 2024-09-11T14:35:55.642 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'tag': None, 'wait_for_package': False} 2024-09-11T14:35:55.642 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:35:55.788 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/ 2024-09-11T14:35:55.788 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4872.g0d61375b 2024-09-11T14:35:56.024 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T14:35:56.024 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:35:56.024 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T14:35:56.052 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T14:35:56.053 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0d61375b8ab53be4c233f5b93e463edfdff9316a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T14:35:56.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T14:35:56.197 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T14:35:56.264 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-09-11T14:35:56.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-09-11T14:35:56.510 INFO:teuthology.orchestra.run.smithi059.stdout:45 files removed 2024-09-11T14:35:56.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2024-09-11T14:35:57.373 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 164 kB/s | 89 kB 00:00 2024-09-11T14:35:57.923 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 48 kB/s | 24 kB 00:00 2024-09-11T14:35:58.448 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 4.3 kB/s | 2.1 kB 00:00 2024-09-11T14:35:58.915 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-11T14:36:00.708 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-11T14:36:06.459 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-11T14:36:08.134 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 19 kB 00:00 2024-09-11T14:36:09.022 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-11T14:36:14.686 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-11T14:36:17.140 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:36:17.151 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:36:17.151 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:36:17.151 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 10 M 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.8 M 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.1 M 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 20 M 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 25 k 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T14:36:17.152 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 808 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 558 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.4 M 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T14:36:17.153 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 45 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 143 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 159 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 316 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 303 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 99 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T14:36:17.154 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T14:36:17.155 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-09-11T14:36:17.155 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T14:36:17.155 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:17.155 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:36:17.155 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:36:17.155 INFO:teuthology.orchestra.run.smithi059.stdout:Install 31 Packages 2024-09-11T14:36:17.155 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2024-09-11T14:36:17.155 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:17.156 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 60 M 2024-09-11T14:36:17.156 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:36:18.148 INFO:teuthology.orchestra.run.smithi059.stdout:(1/33): ceph-base-19.3.0-4872.g0d61375b.el9.x86 6.6 MB/s | 6.1 MB 00:00 2024-09-11T14:36:18.223 INFO:teuthology.orchestra.run.smithi059.stdout:(2/33): ceph-selinux-19.3.0-4872.g0d61375b.el9. 336 kB/s | 25 kB 00:00 2024-09-11T14:36:18.448 INFO:teuthology.orchestra.run.smithi059.stdout:(3/33): libcephfs2-19.3.0-4872.g0d61375b.el9.x8 3.5 MB/s | 808 kB 00:00 2024-09-11T14:36:18.507 INFO:teuthology.orchestra.run.smithi059.stdout:(4/33): libradosstriper1-19.3.0-4872.g0d61375b. 9.4 MB/s | 558 kB 00:00 2024-09-11T14:36:18.782 INFO:teuthology.orchestra.run.smithi059.stdout:(5/33): ceph-common-19.3.0-4872.g0d61375b.el9.x 13 MB/s | 20 MB 00:01 2024-09-11T14:36:18.832 INFO:teuthology.orchestra.run.smithi059.stdout:(6/33): python3-ceph-argparse-19.3.0-4872.g0d61 909 kB/s | 45 kB 00:00 2024-09-11T14:36:18.907 INFO:teuthology.orchestra.run.smithi059.stdout:(7/33): librgw2-19.3.0-4872.g0d61375b.el9.x86_6 13 MB/s | 5.4 MB 00:00 2024-09-11T14:36:18.933 INFO:teuthology.orchestra.run.smithi059.stdout:(8/33): python3-ceph-common-19.3.0-4872.g0d6137 1.4 MB/s | 143 kB 00:00 2024-09-11T14:36:18.958 INFO:teuthology.orchestra.run.smithi059.stdout:(9/33): python3-cephfs-19.3.0-4872.g0d61375b.el 3.1 MB/s | 159 kB 00:00 2024-09-11T14:36:18.991 INFO:teuthology.orchestra.run.smithi059.stdout:(10/33): python3-rados-19.3.0-4872.g0d61375b.el 5.3 MB/s | 316 kB 00:00 2024-09-11T14:36:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:(11/33): python3-rbd-19.3.0-4872.g0d61375b.el9. 5.1 MB/s | 303 kB 00:00 2024-09-11T14:36:19.042 INFO:teuthology.orchestra.run.smithi059.stdout:(12/33): python3-rgw-19.3.0-4872.g0d61375b.el9. 1.9 MB/s | 99 kB 00:00 2024-09-11T14:36:19.167 INFO:teuthology.orchestra.run.smithi059.stdout:(13/33): ceph-radosgw-19.3.0-4872.g0d61375b.el9 5.4 MB/s | 10 MB 00:01 2024-09-11T14:36:19.226 INFO:teuthology.orchestra.run.smithi059.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 512 kB/s | 107 kB 00:00 2024-09-11T14:36:19.268 INFO:teuthology.orchestra.run.smithi059.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-11T14:36:19.293 INFO:teuthology.orchestra.run.smithi059.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 680 kB/s | 171 kB 00:00 2024-09-11T14:36:19.318 INFO:teuthology.orchestra.run.smithi059.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-11T14:36:19.352 INFO:teuthology.orchestra.run.smithi059.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-11T14:36:19.377 INFO:teuthology.orchestra.run.smithi059.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 720 kB/s | 42 kB 00:00 2024-09-11T14:36:19.436 INFO:teuthology.orchestra.run.smithi059.stdout:(20/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-11T14:36:19.461 INFO:teuthology.orchestra.run.smithi059.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 430 kB/s | 36 kB 00:00 2024-09-11T14:36:19.486 INFO:teuthology.orchestra.run.smithi059.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-09-11T14:36:19.512 INFO:teuthology.orchestra.run.smithi059.stdout:(23/33): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2024-09-11T14:36:19.570 INFO:teuthology.orchestra.run.smithi059.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-11T14:36:19.604 INFO:teuthology.orchestra.run.smithi059.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-09-11T14:36:19.629 INFO:teuthology.orchestra.run.smithi059.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 257 kB/s | 26 kB 00:00 2024-09-11T14:36:19.646 INFO:teuthology.orchestra.run.smithi059.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-11T14:36:19.671 INFO:teuthology.orchestra.run.smithi059.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 7.6 MB/s | 191 kB 00:00 2024-09-11T14:36:19.721 INFO:teuthology.orchestra.run.smithi059.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-11T14:36:19.763 INFO:teuthology.orchestra.run.smithi059.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-11T14:36:19.847 INFO:teuthology.orchestra.run.smithi059.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-11T14:36:20.130 INFO:teuthology.orchestra.run.smithi059.stdout:(32/33): librados2-19.3.0-4872.g0d61375b.el9.x8 8.5 MB/s | 3.5 MB 00:00 2024-09-11T14:36:20.247 INFO:teuthology.orchestra.run.smithi059.stdout:(33/33): librbd1-19.3.0-4872.g0d61375b.el9.x86_ 7.8 MB/s | 3.8 MB 00:00 2024-09-11T14:36:20.252 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:36:20.252 INFO:teuthology.orchestra.run.smithi059.stdout:Total 20 MB/s | 60 MB 00:03 2024-09-11T14:36:20.363 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:36:20.404 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:36:20.404 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:36:21.146 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:36:21.147 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:36:22.691 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:36:23.215 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-11T14:36:23.266 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:36:23.359 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:36:23.451 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:36:23.476 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:36:23.539 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:36:23.592 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-11T14:36:23.622 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-11T14:36:23.647 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-11T14:36:23.670 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:36:23.702 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:36:23.888 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:36:23.920 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:36:23.955 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-11T14:36:23.999 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 11/35 2024-09-11T14:36:24.028 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:36:24.129 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-11T14:36:24.158 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:36:24.247 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:36:24.281 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-11T14:36:24.394 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-11T14:36:24.441 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-11T14:36:24.483 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-11T14:36:24.514 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 19/35 2024-09-11T14:36:24.543 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-11T14:36:24.580 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-11T14:36:24.954 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-11T14:36:25.005 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:36:25.081 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:36:25.111 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 24/35 2024-09-11T14:36:25.156 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-11T14:36:25.790 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-11T14:36:25.817 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:36:25.888 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:36:25.969 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 28/35 2024-09-11T14:36:25.994 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 29/35 2024-09-11T14:36:28.601 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:36:28.648 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:36:29.383 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:36:29.418 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:36:29.511 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:36:29.511 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T14:36:29.511 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T14:36:29.511 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:29.544 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:36:40.306 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:36:40.306 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-09-11T14:36:40.307 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-09-11T14:36:40.307 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-09-11T14:36:40.307 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-09-11T14:36:40.307 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-09-11T14:36:40.307 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-09-11T14:36:40.307 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-09-11T14:36:40.307 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:41.459 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:36:41.485 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:36:41.486 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:36:41.486 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T14:36:41.486 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:36:41.486 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:36:41.486 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:41.487 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:36:41.578 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:36:41.578 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:36:42.889 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:36:42.889 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 1/35 2024-09-11T14:36:42.889 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:36:42.889 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:36:42.889 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:36:42.889 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 5/35 2024-09-11T14:36:42.889 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 6/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 7/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 8/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 9/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 10/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 11/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 13/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-11T14:36:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-11T14:36:42.891 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-11T14:36:42.892 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-11T14:36:42.892 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-11T14:36:42.892 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-11T14:36:42.892 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:36:42.892 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-11T14:36:42.892 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 34/35 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.901 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T14:36:43.902 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:43.903 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:36:44.234 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2024-09-11T14:36:44.828 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:30 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:36:44.976 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:36:44.977 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:36:44.977 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:36:44.977 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:36:44.977 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:36:44.977 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 85 M 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 175 k 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout:Install 9 Packages 2024-09-11T14:36:44.978 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:36:44.979 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 87 M 2024-09-11T14:36:44.979 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 797 M 2024-09-11T14:36:44.979 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:36:45.340 INFO:teuthology.orchestra.run.smithi059.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 623 kB/s | 94 kB 00:00 2024-09-11T14:36:45.449 INFO:teuthology.orchestra.run.smithi059.stdout:(2/9): libcephsqlite-19.3.0-4872.g0d61375b.el9. 673 kB/s | 175 kB 00:00 2024-09-11T14:36:45.999 INFO:teuthology.orchestra.run.smithi059.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 332 kB/s | 183 kB 00:00 2024-09-11T14:36:46.091 INFO:teuthology.orchestra.run.smithi059.stdout:(4/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 684 kB/s | 513 kB 00:00 2024-09-11T14:36:46.208 INFO:teuthology.orchestra.run.smithi059.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-11T14:36:46.317 INFO:teuthology.orchestra.run.smithi059.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 588 kB/s | 64 kB 00:00 2024-09-11T14:36:46.359 INFO:teuthology.orchestra.run.smithi059.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.9 MB/s | 122 kB 00:00 2024-09-11T14:36:46.551 INFO:teuthology.orchestra.run.smithi059.stdout:(8/9): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-09-11T14:36:47.984 INFO:teuthology.orchestra.run.smithi059.stdout:(9/9): ceph-test-19.3.0-4872.g0d61375b.el9.x86_ 30 MB/s | 85 MB 00:02 2024-09-11T14:36:47.987 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:36:47.988 INFO:teuthology.orchestra.run.smithi059.stdout:Total 29 MB/s | 87 MB 00:03 2024-09-11T14:36:48.075 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:36:48.101 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:36:48.101 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:36:48.733 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:36:48.734 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:36:49.647 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:36:49.716 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-11T14:36:49.762 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-11T14:36:49.806 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-11T14:36:49.837 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-11T14:36:49.871 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:36:49.895 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:36:49.957 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:36:50.005 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-11T14:36:50.039 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:37:01.969 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:37:02.048 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 1/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-11T14:37:07.934 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T14:37:08.335 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T14:37:08.336 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T14:37:08.336 INFO:teuthology.orchestra.run.smithi059.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-11T14:37:08.336 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:08.336 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:37:08.678 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2024-09-11T14:37:09.264 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:37:09.406 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:37:09.409 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:37:09.409 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:09.409 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:37:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.5 k 2024-09-11T14:37:09.409 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:37:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 2.2 M 2024-09-11T14:37:09.409 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 1.6 M 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 291 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.1 M 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 71 M 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T14:37:09.410 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T14:37:09.411 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout:Install 37 Packages 2024-09-11T14:37:09.412 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:09.413 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 84 M 2024-09-11T14:37:09.413 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 896 M 2024-09-11T14:37:09.413 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:37:10.429 INFO:teuthology.orchestra.run.smithi059.stdout:(1/37): ceph-19.3.0-4872.g0d61375b.el9.x86_64.r 32 kB/s | 6.5 kB 00:00 2024-09-11T14:37:10.605 INFO:teuthology.orchestra.run.smithi059.stdout:(2/37): ceph-mgr-19.3.0-4872.g0d61375b.el9.x86_ 4.2 MB/s | 1.6 MB 00:00 2024-09-11T14:37:10.655 INFO:teuthology.orchestra.run.smithi059.stdout:(3/37): ceph-mds-19.3.0-4872.g0d61375b.el9.x86_ 5.1 MB/s | 2.2 MB 00:00 2024-09-11T14:37:10.755 INFO:teuthology.orchestra.run.smithi059.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4872.g0d61 2.8 MB/s | 291 kB 00:00 2024-09-11T14:37:10.830 INFO:teuthology.orchestra.run.smithi059.stdout:(5/37): ceph-mon-19.3.0-4872.g0d61375b.el9.x86_ 13 MB/s | 5.1 MB 00:00 2024-09-11T14:37:11.031 INFO:teuthology.orchestra.run.smithi059.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 921 kB/s | 253 kB 00:00 2024-09-11T14:37:11.089 INFO:teuthology.orchestra.run.smithi059.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-11T14:37:11.148 INFO:teuthology.orchestra.run.smithi059.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-11T14:37:11.190 INFO:teuthology.orchestra.run.smithi059.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.4 MB/s | 1.2 MB 00:00 2024-09-11T14:37:11.215 INFO:teuthology.orchestra.run.smithi059.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-11T14:37:11.249 INFO:teuthology.orchestra.run.smithi059.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-11T14:37:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-09-11T14:37:11.416 INFO:teuthology.orchestra.run.smithi059.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-11T14:37:11.433 INFO:teuthology.orchestra.run.smithi059.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 834 kB/s | 35 kB 00:00 2024-09-11T14:37:11.466 INFO:teuthology.orchestra.run.smithi059.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-11T14:37:11.483 INFO:teuthology.orchestra.run.smithi059.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-11T14:37:11.508 INFO:teuthology.orchestra.run.smithi059.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-11T14:37:11.551 INFO:teuthology.orchestra.run.smithi059.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-09-11T14:37:11.567 INFO:teuthology.orchestra.run.smithi059.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.6 MB/s | 43 kB 00:00 2024-09-11T14:37:11.593 INFO:teuthology.orchestra.run.smithi059.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.8 MB/s | 172 kB 00:00 2024-09-11T14:37:11.626 INFO:teuthology.orchestra.run.smithi059.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-11T14:37:11.643 INFO:teuthology.orchestra.run.smithi059.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 646 kB/s | 11 kB 00:00 2024-09-11T14:37:11.660 INFO:teuthology.orchestra.run.smithi059.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-11T14:37:11.677 INFO:teuthology.orchestra.run.smithi059.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-09-11T14:37:11.694 INFO:teuthology.orchestra.run.smithi059.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 1.1 MB/s | 19 kB 00:00 2024-09-11T14:37:11.711 INFO:teuthology.orchestra.run.smithi059.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 1.2 MB/s | 20 kB 00:00 2024-09-11T14:37:11.736 INFO:teuthology.orchestra.run.smithi059.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-11T14:37:11.761 INFO:teuthology.orchestra.run.smithi059.stdout:(28/37): python3-more-itertools-8.12.0-2.el9.no 3.1 MB/s | 79 kB 00:00 2024-09-11T14:37:11.787 INFO:teuthology.orchestra.run.smithi059.stdout:(29/37): lua-devel-5.4.4-4.el9.x86_64.rpm 73 kB/s | 22 kB 00:00 2024-09-11T14:37:11.812 INFO:teuthology.orchestra.run.smithi059.stdout:(30/37): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-11T14:37:11.846 INFO:teuthology.orchestra.run.smithi059.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-11T14:37:11.871 INFO:teuthology.orchestra.run.smithi059.stdout:(32/37): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-09-11T14:37:11.896 INFO:teuthology.orchestra.run.smithi059.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 746 kB/s | 36 kB 00:00 2024-09-11T14:37:11.930 INFO:teuthology.orchestra.run.smithi059.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 6.8 MB/s | 230 kB 00:00 2024-09-11T14:37:11.955 INFO:teuthology.orchestra.run.smithi059.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-09-11T14:37:11.997 INFO:teuthology.orchestra.run.smithi059.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-11T14:37:12.964 INFO:teuthology.orchestra.run.smithi059.stdout:(37/37): ceph-osd-19.3.0-4872.g0d61375b.el9.x86 30 MB/s | 71 MB 00:02 2024-09-11T14:37:12.968 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:37:12.968 INFO:teuthology.orchestra.run.smithi059.stdout:Total 24 MB/s | 84 MB 00:03 2024-09-11T14:37:13.125 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:37:13.175 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:37:13.175 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:37:13.984 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:37:13.984 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:37:15.665 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:37:15.716 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T14:37:15.750 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T14:37:15.777 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T14:37:15.852 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T14:37:15.897 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T14:37:15.929 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T14:37:16.025 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T14:37:16.111 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T14:37:16.157 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T14:37:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T14:37:16.221 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T14:37:16.255 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T14:37:16.289 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T14:37:16.330 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T14:37:16.373 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T14:37:16.407 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T14:37:16.457 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T14:37:16.496 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T14:37:16.572 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T14:37:16.740 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T14:37:16.824 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T14:37:16.874 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T14:37:16.912 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T14:37:16.970 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T14:37:17.043 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T14:37:17.240 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T14:37:17.291 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T14:37:17.372 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T14:37:17.477 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T14:37:17.514 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T14:37:17.658 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T14:37:17.859 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 32/37 2024-09-11T14:37:17.898 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:37:17.925 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:37:17.925 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:37:17.925 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T14:37:17.925 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:37:17.926 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:37:17.926 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:29.098 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:37:29.127 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:37:29.127 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:37:29.127 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T14:37:29.128 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:37:29.128 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:37:29.128 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:29.708 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:37:29.735 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:37:29.735 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:37:29.735 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T14:37:29.735 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:37:29.735 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:37:29.735 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:30.017 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:37:30.042 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:37:30.042 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:37:30.043 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T14:37:30.043 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:37:30.043 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:37:30.043 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:30.108 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:37:36.561 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:37:36.561 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 1/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 3/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 4/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 5/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 6/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T14:37:36.562 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T14:37:36.563 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T14:37:36.565 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T14:37:37.530 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T14:37:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T14:37:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T14:37:37.533 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T14:37:37.534 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:37.534 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:37:37.773 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2024-09-11T14:37:38.355 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:37:38.448 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:37:38.493 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:37:38.495 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:37:38.495 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:37:38.560 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2024-09-11T14:37:39.138 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:25 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:37:39.276 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 784 k 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 784 k 2024-09-11T14:37:39.277 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 791 k 2024-09-11T14:37:39.278 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:37:39.549 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-19.3.0-4872.g0d61375b.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-11T14:37:39.549 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:37:39.549 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-11T14:37:39.549 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:37:39.555 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:37:39.555 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:37:39.561 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:37:39.562 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:37:39.644 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:37:40.112 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:37:40.221 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:37:40.566 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:37:40.823 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:37:40.824 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:40.824 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:37:40.824 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:37:40.824 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:40.824 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:37:41.009 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T14:37:41.598 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:37:41.736 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:37:41.737 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:41.737 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-09-11T14:37:41.737 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:41.737 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:37:41.737 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 158 k 2024-09-11T14:37:41.737 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:41.738 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:37:41.738 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:41.738 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-11T14:37:41.738 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:41.738 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 158 k 2024-09-11T14:37:41.738 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 644 k 2024-09-11T14:37:41.738 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:37:41.914 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-19.3.0-4872.g0d6137 894 kB/s | 158 kB 00:00 2024-09-11T14:37:41.915 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:37:41.915 INFO:teuthology.orchestra.run.smithi059.stdout:Total 885 kB/s | 158 kB 00:00 2024-09-11T14:37:41.916 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:37:41.926 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:37:41.926 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:37:41.988 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:37:41.988 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:37:42.150 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:37:42.223 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:37:42.250 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:37:42.250 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:37:42.250 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T14:37:42.250 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:43.183 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:37:43.183 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:43.183 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:37:43.183 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:37:43.183 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:43.183 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:37:43.384 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2024-09-11T14:37:43.970 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:29 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:37:44.063 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:37:44.108 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:37:44.109 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:37:44.109 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:37:44.171 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T14:37:44.761 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:30 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:37:44.900 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:37:44.901 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:44.901 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:37:44.901 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:44.901 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:37:44.901 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 14 M 2024-09-11T14:37:44.901 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:37:44.901 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 34 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 16 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout:Install 11 Packages 2024-09-11T14:37:44.902 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:44.903 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 18 M 2024-09-11T14:37:44.903 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 155 M 2024-09-11T14:37:44.903 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:37:45.742 INFO:teuthology.orchestra.run.smithi059.stdout:(1/11): ceph-prometheus-alerts-19.3.0-4872.g0d6 56 kB/s | 16 kB 00:00 2024-09-11T14:37:45.767 INFO:teuthology.orchestra.run.smithi059.stdout:(2/11): ceph-grafana-dashboards-19.3.0-4872.g0d 107 kB/s | 34 kB 00:00 2024-09-11T14:37:45.976 INFO:teuthology.orchestra.run.smithi059.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-09-11T14:37:46.026 INFO:teuthology.orchestra.run.smithi059.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-11T14:37:46.051 INFO:teuthology.orchestra.run.smithi059.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 775 kB/s | 19 kB 00:00 2024-09-11T14:37:46.127 INFO:teuthology.orchestra.run.smithi059.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 27 MB/s | 2.0 MB 00:00 2024-09-11T14:37:46.152 INFO:teuthology.orchestra.run.smithi059.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-11T14:37:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-11T14:37:46.202 INFO:teuthology.orchestra.run.smithi059.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-11T14:37:46.511 INFO:teuthology.orchestra.run.smithi059.stdout:(10/11): ceph-mgr-dashboard-19.3.0-4872.g0d6137 13 MB/s | 14 MB 00:01 2024-09-11T14:37:46.628 INFO:teuthology.orchestra.run.smithi059.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-09-11T14:37:46.632 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:37:46.632 INFO:teuthology.orchestra.run.smithi059.stdout:Total 11 MB/s | 18 MB 00:01 2024-09-11T14:37:46.666 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:37:46.688 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:37:46.689 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:37:46.970 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:37:46.970 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:37:47.530 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:37:47.611 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-11T14:37:47.640 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-11T14:37:47.746 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-11T14:37:47.881 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-11T14:37:47.926 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-11T14:37:48.013 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-11T14:37:48.179 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-11T14:37:48.206 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:37:48.239 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 9/11 2024-09-11T14:37:51.619 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 10/11 2024-09-11T14:37:51.676 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:37:53.375 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 1/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 2/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 3/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-11T14:37:53.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-11T14:37:53.689 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-11T14:37:53.689 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T14:37:53.690 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:53.691 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:37:53.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T14:37:54.519 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:37:54.660 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 7.4 M 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T14:37:54.661 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2024-09-11T14:37:54.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:37:54.663 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 38 M 2024-09-11T14:37:54.663 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 212 M 2024-09-11T14:37:54.663 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:37:55.150 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-11T14:37:55.200 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 654 kB/s | 33 kB 00:00 2024-09-11T14:37:55.267 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-11T14:37:55.317 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-11T14:37:55.417 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-11T14:37:55.442 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 307 kB/s | 38 kB 00:00 2024-09-11T14:37:55.501 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-11T14:37:55.568 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.1 MB/s | 3.0 MB 00:00 2024-09-11T14:37:55.627 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-11T14:37:55.969 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-09-11T14:37:56.094 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 7.3 MB/s | 4.9 MB 00:00 2024-09-11T14:37:56.511 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-09-11T14:37:56.511 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:37:56.512 INFO:teuthology.orchestra.run.smithi059.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-11T14:37:56.876 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:37:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:37:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:37:57.368 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:37:57.368 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:37:58.157 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:37:58.232 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T14:37:58.268 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:37:58.371 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:37:58.410 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T14:37:58.601 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T14:37:58.634 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T14:37:58.712 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T14:37:59.392 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T14:37:59.516 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T14:38:00.832 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:38:01.906 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T14:38:01.932 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:38:04.449 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:38:04.449 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 1/12 2024-09-11T14:38:04.449 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:38:04.450 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T14:38:04.858 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T14:38:04.858 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:04.858 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:04.858 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:38:04.858 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T14:38:04.858 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T14:38:04.858 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T14:38:04.858 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:04.859 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:05.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2024-09-11T14:38:05.806 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:05.947 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:05.948 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 49 k 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T14:38:05.949 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2024-09-11T14:38:05.950 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:05.951 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.2 M 2024-09-11T14:38:05.951 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2024-09-11T14:38:05.951 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:06.440 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 196 kB/s | 19 kB 00:00 2024-09-11T14:38:06.465 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 26 kB 00:00 2024-09-11T14:38:06.524 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): ceph-mgr-rook-19.3.0-4872.g0d61375b.el9 271 kB/s | 49 kB 00:00 2024-09-11T14:38:06.566 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-11T14:38:06.591 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-11T14:38:06.616 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-11T14:38:06.658 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-11T14:38:06.684 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-11T14:38:06.709 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-11T14:38:06.743 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-11T14:38:06.768 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-11T14:38:06.793 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-09-11T14:38:06.826 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-11T14:38:06.830 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:06.830 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-11T14:38:06.903 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:06.920 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:06.920 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:07.018 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:07.019 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:07.528 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:07.706 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T14:38:07.754 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T14:38:07.810 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T14:38:07.846 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T14:38:07.888 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T14:38:07.993 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T14:38:08.120 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T14:38:08.181 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T14:38:08.939 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T14:38:08.994 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T14:38:09.028 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T14:38:09.077 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T14:38:09.097 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:38:09.772 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:38:09.772 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 1/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T14:38:09.773 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T14:38:10.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T14:38:10.122 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T14:38:10.123 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T14:38:10.123 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T14:38:10.123 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T14:38:10.123 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:10.123 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:10.365 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T14:38:10.959 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:56 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:11.101 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:11.101 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:11.101 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 154 k 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:11.102 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2024-09-11T14:38:11.103 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:11.103 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 7.0 M 2024-09-11T14:38:11.103 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 33 M 2024-09-11T14:38:11.103 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:11.386 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-11T14:38:11.436 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-11T14:38:11.461 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-11T14:38:11.487 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-11T14:38:11.511 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): ceph-mgr-cephadm-19.3.0-4872.g0d61375b.e 478 kB/s | 154 kB 00:00 2024-09-11T14:38:12.229 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.7 MB/s | 6.0 MB 00:01 2024-09-11T14:38:12.231 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:12.231 INFO:teuthology.orchestra.run.smithi059.stdout:Total 6.2 MB/s | 7.0 MB 00:01 2024-09-11T14:38:12.305 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:12.317 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:12.317 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:12.424 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:12.425 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:12.753 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:12.953 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T14:38:12.998 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T14:38:13.552 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T14:38:13.653 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T14:38:13.748 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T14:38:13.775 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:38:14.597 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:38:14.597 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/6 2024-09-11T14:38:14.597 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T14:38:14.597 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T14:38:14.597 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T14:38:14.597 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T14:38:14.878 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T14:38:14.878 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:14.878 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:14.878 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:38:14.878 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T14:38:14.878 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T14:38:14.879 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T14:38:14.879 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T14:38:14.879 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T14:38:14.879 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:14.879 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:15.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2024-09-11T14:38:15.705 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:01 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:15.844 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:15.845 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:15.845 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:38:15.845 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:15.845 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 884 k 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 964 k 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 4.1 M 2024-09-11T14:38:15.846 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:16.697 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): ceph-fuse-19.3.0-4872.g0d61375b.el9.x86_ 1.5 MB/s | 884 kB 00:00 2024-09-11T14:38:16.814 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 112 kB/s | 80 kB 00:00 2024-09-11T14:38:16.815 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:16.816 INFO:teuthology.orchestra.run.smithi059.stdout:Total 994 kB/s | 964 kB 00:00 2024-09-11T14:38:16.829 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:16.857 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:16.857 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:16.945 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:16.945 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:17.138 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:17.280 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T14:38:17.358 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:38:18.043 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:38:18.043 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/2 2024-09-11T14:38:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T14:38:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:18.320 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:18.320 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T14:38:18.321 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:18.321 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:18.624 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-volume 2024-09-11T14:38:19.221 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:05 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:19.360 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:19.361 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:19.361 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-09-11T14:38:19.361 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:19.361 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:19.361 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 285 k 2024-09-11T14:38:19.361 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:19.362 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:19.362 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:19.362 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-11T14:38:19.362 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:19.362 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 285 k 2024-09-11T14:38:19.362 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.3 M 2024-09-11T14:38:19.362 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:19.851 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-volume-19.3.0-4872.g0d61375b.el9.noarch.rp 583 kB/s | 285 kB 00:00 2024-09-11T14:38:19.852 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:19.852 INFO:teuthology.orchestra.run.smithi059.stdout:Total 581 kB/s | 285 kB 00:00 2024-09-11T14:38:19.852 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:19.862 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:19.862 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:19.892 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:19.892 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:20.126 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:20.195 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:38:20.212 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:38:20.212 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:38:20.212 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T14:38:20.212 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:21.227 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:38:21.227 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:21.227 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:21.227 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:38:21.227 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:21.228 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:21.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2024-09-11T14:38:22.079 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:08 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:22.218 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:22.219 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:22.219 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-09-11T14:38:22.219 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:22.219 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:22.219 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 125 k 2024-09-11T14:38:22.220 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:22.220 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:22.220 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:22.220 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-11T14:38:22.220 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:22.220 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 125 k 2024-09-11T14:38:22.220 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 445 k 2024-09-11T14:38:22.220 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:22.598 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-19.3.0-4872.g0d61375b.el9.x86_64 330 kB/s | 125 kB 00:00 2024-09-11T14:38:22.598 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:22.599 INFO:teuthology.orchestra.run.smithi059.stdout:Total 329 kB/s | 125 kB 00:00 2024-09-11T14:38:22.599 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:22.604 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:22.604 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:22.673 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:22.673 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:22.835 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:22.949 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:23.374 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:23.655 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:23.655 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:23.655 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:23.656 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:23.656 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:23.656 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:23.899 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2024-09-11T14:38:24.497 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:10 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:24.591 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:38:24.639 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:24.641 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:38:24.641 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:24.706 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2024-09-11T14:38:25.296 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:11 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:25.434 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:25.435 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:25.435 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-09-11T14:38:25.435 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:25.435 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 32 k 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 32 k 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 142 k 2024-09-11T14:38:25.436 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:25.717 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-19.3.0-4872.g0d61375b.el9.x86_6 113 kB/s | 32 kB 00:00 2024-09-11T14:38:25.718 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:25.718 INFO:teuthology.orchestra.run.smithi059.stdout:Total 112 kB/s | 32 kB 00:00 2024-09-11T14:38:25.718 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:25.722 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:25.722 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:25.739 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:25.740 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:25.829 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:25.951 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:26.334 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:26.576 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:26.576 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:26.577 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:26.577 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:26.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2024-09-11T14:38:27.357 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:13 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:27.451 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:38:27.497 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:27.499 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:38:27.499 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:27.571 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2024-09-11T14:38:28.161 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:14 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:28.254 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:38:28.300 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:28.301 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:38:28.301 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:28.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2024-09-11T14:38:28.957 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:14 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:29.050 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:38:29.098 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:29.099 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:38:29.099 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:29.171 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2024-09-11T14:38:29.764 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:15 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:29.859 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:38:29.907 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:29.908 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:38:29.908 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:29.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2024-09-11T14:38:30.579 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:16 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:30.671 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:38:30.717 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:30.718 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:38:30.718 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:30.784 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2024-09-11T14:38:31.373 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:17 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:31.465 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:38:31.511 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:31.512 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2024-09-11T14:38:31.512 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:31.578 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2024-09-11T14:38:32.169 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:18 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:32.308 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:32.309 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:32.309 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-09-11T14:38:32.309 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 91 k 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 91 k 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 321 k 2024-09-11T14:38:32.310 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:32.687 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-19.3.0-4872.g0d61375b.el9.x86_64.rpm 241 kB/s | 91 kB 00:00 2024-09-11T14:38:32.688 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:32.688 INFO:teuthology.orchestra.run.smithi059.stdout:Total 240 kB/s | 91 kB 00:00 2024-09-11T14:38:32.688 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:32.698 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:32.698 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:32.766 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:32.766 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:32.920 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:33.037 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:33.431 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:33.686 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:33.686 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:33.686 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:33.686 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:33.686 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:33.686 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:33.924 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2024-09-11T14:38:34.531 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:20 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:34.677 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-11T14:38:34.678 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:34.679 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.5 M 2024-09-11T14:38:34.679 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 30 M 2024-09-11T14:38:34.679 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:35.865 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-19.3.0-4872.g0d61375b.el9.x86_64.rpm 3.0 MB/s | 3.5 MB 00:01 2024-09-11T14:38:35.865 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:35.866 INFO:teuthology.orchestra.run.smithi059.stdout:Total 3.0 MB/s | 3.5 MB 00:01 2024-09-11T14:38:35.866 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:35.880 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:35.880 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:35.970 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:35.970 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:36.582 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:36.640 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:36.667 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:36.667 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:38:36.667 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T14:38:36.667 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:38:36.668 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:38:36.668 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:37.889 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:37.890 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:37.890 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:37.890 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:37.890 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:37.890 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:38.126 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2024-09-11T14:38:38.719 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:24 ago on Wed 11 Sep 2024 02:36:14 PM UTC. 2024-09-11T14:38:38.860 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 177 k 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 177 k 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 685 k 2024-09-11T14:38:38.861 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-09-11T14:38:39.351 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-19.3.0-4872.g0d61375b.el9.x86_64.rpm 362 kB/s | 177 kB 00:00 2024-09-11T14:38:39.351 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:38:39.351 INFO:teuthology.orchestra.run.smithi059.stdout:Total 360 kB/s | 177 kB 00:00 2024-09-11T14:38:39.352 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-09-11T14:38:39.363 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-09-11T14:38:39.363 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-09-11T14:38:39.431 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-09-11T14:38:39.432 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-09-11T14:38:39.597 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-09-11T14:38:39.760 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:40.185 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:40.466 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:38:40.467 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:40.467 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-09-11T14:38:40.467 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:38:40.467 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:40.467 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-09-11T14:38:40.722 DEBUG:teuthology.parallel:result is None 2024-09-11T14:38:40.723 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:38:40.864 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T14:38:40.886 INFO:teuthology.orchestra.run.smithi059.stdout:19.3.0-4872.g0d61375b.el9 2024-09-11T14:38:40.887 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4872.g0d61375b.el9 2024-09-11T14:38:40.887 INFO:teuthology.task.install:The correct ceph version 19.3.0-4872.g0d61375b is installed. 2024-09-11T14:38:40.889 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T14:38:40.889 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:40.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T14:38:40.954 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T14:38:40.954 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:40.954 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T14:38:41.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T14:38:41.085 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T14:38:41.085 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:41.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T14:38:41.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T14:38:41.218 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T14:38:41.218 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:41.218 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T14:38:41.284 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T14:38:41.349 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T14:38:41.434 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T14:38:41.434 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-09-11T14:38:41.461 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T14:38:41.461 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T14:38:41.526 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T14:38:41.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T14:38:41.598 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T14:38:41.598 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T14:38:41.598 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901370', 'branch': 'main', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7901370', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'workunit': {'branch': 'wip-matanb-crimson-seastore-logs', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5813, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-seastore-logs', 'suite_path': '/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/matan-b/ceph.git', 'suite_sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2', 'targets': {'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHrT/poIiaUu1xUmApBoEuHf1hr2WTVqxSj2beq/agv8s9W41ud2Wwvw868ZO18APM9N7E52uFZXRg/X3QVT6Uo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}, 'env': {'RBD_FEATURES': '61'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-11_13:56:22', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T14:38:41.598 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T14:38:41.648 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T14:38:41.713 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:41.714 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T14:38:41.770 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T14:38:41.770 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-09-11T14:38:41.826 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T14:38:41.826 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:38:41.826 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-11T14:38:41.826 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:38:41.826 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:38:41.826 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-11 14:38:36.951390645 +0000 2024-09-11T14:38:41.827 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-11 14:35:36.289885144 +0000 2024-09-11T14:38:41.827 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-11 14:35:36.289885144 +0000 2024-09-11T14:38:41.827 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-09-11 14:35:36.289885144 +0000 2024-09-11T14:38:41.827 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T14:38:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-11T14:38:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-11T14:38:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000188307 s, 2.7 MB/s 2024-09-11T14:38:41.893 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T14:38:41.951 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-09-11T14:38:42.006 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T14:38:42.007 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:38:42.007 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-11T14:38:42.007 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:38:42.007 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:38:42.007 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-11 14:38:36.951390645 +0000 2024-09-11T14:38:42.007 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-11 14:35:36.616880822 +0000 2024-09-11T14:38:42.007 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-11 14:35:36.616880822 +0000 2024-09-11T14:38:42.007 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-09-11 14:35:36.616880822 +0000 2024-09-11T14:38:42.007 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T14:38:42.071 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-11T14:38:42.071 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-11T14:38:42.071 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.00019504 s, 2.6 MB/s 2024-09-11T14:38:42.072 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T14:38:42.129 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-09-11T14:38:42.184 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T14:38:42.184 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:38:42.185 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-11T14:38:42.185 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:38:42.185 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:38:42.185 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-11 14:38:36.952390631 +0000 2024-09-11T14:38:42.185 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-11 14:35:36.957876314 +0000 2024-09-11T14:38:42.185 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-11 14:35:36.957876314 +0000 2024-09-11T14:38:42.185 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-09-11 14:35:36.957876314 +0000 2024-09-11T14:38:42.185 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T14:38:42.249 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-11T14:38:42.249 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-11T14:38:42.249 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000207828 s, 2.5 MB/s 2024-09-11T14:38:42.250 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T14:38:42.307 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-09-11 14:38:36.952390631 +0000 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-09-11 14:35:37.284871992 +0000 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-09-11 14:35:37.284871992 +0000 2024-09-11T14:38:42.363 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-09-11 14:35:37.284871992 +0000 2024-09-11T14:38:42.363 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T14:38:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-09-11T14:38:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-09-11T14:38:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000211038 s, 2.4 MB/s 2024-09-11T14:38:42.429 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T14:38:42.485 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:38:42.486 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T14:38:42.486 INFO:tasks.ceph:Generating config... 2024-09-11T14:38:42.487 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-11T14:38:42.487 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-11T14:38:42.487 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-11T14:38:42.487 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-11T14:38:42.487 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-11T14:38:42.487 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T14:38:42.487 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T14:38:42.488 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T14:38:42.488 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-11T14:38:42.488 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-11T14:38:42.488 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-11T14:38:42.488 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-11T14:38:42.488 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T14:38:42.488 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T14:38:42.488 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-11T14:38:42.488 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-11T14:38:42.489 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-11T14:38:42.490 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-11T14:38:42.490 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:38:42.490 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-11T14:38:42.490 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-11T14:38:42.490 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-11T14:38:42.490 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-11T14:38:42.490 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-11T14:38:42.490 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-11T14:38:42.490 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T14:38:42.490 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T14:38:42.490 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-11T14:38:42.490 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-11T14:38:42.490 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-11T14:38:42.491 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T14:38:42.491 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T14:38:42.581 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T14:38:42.584 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T14:38:42.637 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:38:42.662 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.59')] 2024-09-11T14:38:42.663 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.59', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-11T14:38:42.664 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:42.664 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:38:42.719 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.59 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:38:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:42.810+0000 7fc0f7479f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:42.810+0000 7fc0f7479f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:42.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:42.811+0000 7fc0f7479f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:42.812 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid 1af81b22-8a9d-4464-aa64-8b69c827c577 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = quincy 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:fsid 1af81b22-8a9d-4464-aa64-8b69c827c577 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2024-09-11T14:38:42.812123+0000 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-09-11T14:38:42.812123+0000 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 17 (quincy) 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 1 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2024-09-11T14:38:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-11T14:38:42.814 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:38:42.830 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1af81b22-8a9d-4464-aa64-8b69c827c577... 2024-09-11T14:38:42.831 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T14:38:42.914 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-09-11T14:38:42.915 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-09-11T14:38:42.916 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-09-11T14:38:42.917 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = 172.21.15.59 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: ms client mode = crc 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: ms cluster mode = crc 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: ms mon client mode = crc 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: ms mon cluster mode = crc 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: ms mon service mode = crc 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: ms service mode = crc 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 1af81b22-8a9d-4464-aa64-8b69c827c577 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-09-11T14:38:42.918 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-09-11T14:38:42.919 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: debug monc = 20 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: crimson alien thread cpu cores = 6-7 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: crimson osd obc lru size = 10 2024-09-11T14:38:42.920 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore backref = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore cache = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore cleaner = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore device = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore epm = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore fixedkv tree = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore journal = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore lba = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore odata = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore ompap = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore onode = 20 2024-09-11T14:38:42.921 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore t = 20 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: debug seastore tm = 20 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = seastore 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: osd shutdown pgref assert = True 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.922 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T14:38:42.923 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon min osdmap epochs = 50 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune interval = 2 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune min = 15 2024-09-11T14:38:42.924 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune txsize = 2 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: paxos service trim min = 10 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default crimson = True 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: crimson seastar cpu cores = 0-1 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-09-11T14:38:42.925 INFO:teuthology.orchestra.run.smithi059.stdout: crimson seastar cpu cores = 2-3 2024-09-11T14:38:42.926 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-09-11T14:38:42.926 INFO:teuthology.orchestra.run.smithi059.stdout: crimson seastar cpu cores = 4-5 2024-09-11T14:38:42.926 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T14:38:42.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T14:38:43.034 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T14:38:43.034 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:43.034 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T14:38:43.050 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:43.050 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T14:38:43.106 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T14:38:43.106 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:43.106 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T14:38:43.106 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:38:43.182 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:43.182 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:38:43.238 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T14:38:43.239 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T14:38:43.239 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:38:43.340 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:38:43.343 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T14:38:43.343 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T14:38:43.343 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T14:38:43.394 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T14:38:43.408 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T14:38:43.408 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T14:38:43.408 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T14:38:43.433 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:38:43.433 INFO:tasks.ceph:role: osd.0 2024-09-11T14:38:43.433 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T14:38:43.434 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:38:43.507 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:38:44.013 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-11T14:38:44.015 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-11T14:38:44.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T14:38:44.149 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T14:38:44.185 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T14:38:44.252 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:38:44.252 INFO:tasks.ceph:role: osd.1 2024-09-11T14:38:44.252 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T14:38:44.252 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T14:38:44.326 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:38:44.326 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:38:44.326 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:38:44.326 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:38:44.326 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:38:44.326 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:38:44.326 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:38:44.326 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:38:44.327 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:38:44.327 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:38:44.775 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-11T14:38:44.777 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-11T14:38:44.777 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T14:38:44.863 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T14:38:44.893 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T14:38:44.958 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:38:44.959 INFO:tasks.ceph:role: osd.2 2024-09-11T14:38:44.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T14:38:44.959 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T14:38:45.032 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:38:45.032 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:38:45.032 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:38:45.032 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:38:45.033 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:38:45.033 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:38:45.033 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:38:45.033 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:38:45.033 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:38:45.033 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:38:45.518 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-09-11T14:38:45.521 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-09-11T14:38:45.521 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T14:38:45.663 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T14:38:45.695 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:38:46.011 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:38:46.012 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:46,012 seastar - Reactor backend: linux-aio 2024-09-11T14:38:46.023 INFO:teuthology.orchestra.run.smithi059.stderr:==27906==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:38:46.158 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:46,158 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:38:46.159 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:46,158 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-11T14:38:46.159 INFO:teuthology.orchestra.run.smithi059.stderr:DEBUG 2024-09-11 14:38:46,159 [shard 0:main] seastar - reactor::drain 2024-09-11T14:38:46.714 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:46.714 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-09-11T14:38:46.714 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-09-11T14:38:46.714 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-09-11T14:38:46.715 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:46.715 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-11T14:38:46.773 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:38:46.777 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:46,777 seastar - Reactor backend: linux-aio 2024-09-11T14:38:46.785 INFO:teuthology.orchestra.run.smithi059.stderr:==27904==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:38:47.089 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3642e27 0xba293f0 0xba2964e 0xba2979b 0xb9d1e3e 0xb9d64f1 0x984ae51 0x9b01034 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:47.089 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:47.147 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x955551b 0x95d11f8 0x95d131d 0x95eab49 0x984adb5 0x9b05eb6 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:47.147 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:47.396 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x55494cf 0x9e3b019 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:47.396 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:47.453 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10252c 0x10d76c 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:47.453 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:47.482 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:47.482 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:47.539 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133c2d 0xdd332 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:47.539 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:47.561 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554912b 0x554a11c 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:47.561 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:47.647 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554a048 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:47.647 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:48.101 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x554a0b1 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:48.101 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:48.159 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549a3d 0x9e3b03b 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:48.159 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:49.151 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:49.151 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-09-11T14:38:49.151 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-09-11T14:38:49.151 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-09-11T14:38:49.151 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:49.151 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-11T14:38:49.176 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T14:38:49.202 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:38:49.493 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:38:49.494 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:49,494 seastar - Reactor backend: linux-aio 2024-09-11T14:38:49.500 INFO:teuthology.orchestra.run.smithi059.stderr:==27951==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:38:49.608 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x38e6f1d 0x38e79d7 0x38e7c9f 0x5eeee4b 0x5ee916a 0x5e09eb1 0x5e18896 0x5e1bcac 0x5e1cad6 0x5e1f323 0x5e1ff3a 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x3968bb0 0x396a38a 0x380ac18 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:49.609 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:49.628 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:49,628 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:38:49.629 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:49,628 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-11T14:38:49.629 INFO:teuthology.orchestra.run.smithi059.stderr:DEBUG 2024-09-11 14:38:49,629 [shard 0:main] seastar - reactor::drain 2024-09-11T14:38:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-09-11T14:38:50.157 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-09-11T14:38:50.157 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-09-11T14:38:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:50.157 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-11T14:38:50.215 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:38:50.219 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:50,219 seastar - Reactor backend: linux-aio 2024-09-11T14:38:50.232 INFO:teuthology.orchestra.run.smithi059.stderr:==27949==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:38:50.297 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4ab1c 0x5ff0dd6 0x5ff0f9e 0x5ff11cf 0x5ff14a5 0x603fba3 0x603fda9 0x5fbfef6 0x5fc00d0 0x5fc065c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:50.297 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:50.534 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xfd531 0xb85ab4d 0xb85ac00 0xb85ad16 0xb85adf6 0xb85b076 0xb85b621 0xb85ba2c 0xb85bceb 0xb85c7a8 0xba2d476 0xb9d2631 0x9b01657 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:50.535 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:50.592 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xfa067 0x360e5b6 0x3619d66 0x36301ae 0x38311c8 0x9c23fe2 0x9c24281 0x9c24510 0x9b05d33 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:50.592 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:50.619 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xb9ee553 0xba179db 0xb9d61ef 0x984ae51 0x9b0161a 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:50.619 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:50.676 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133caf 0xdd332 0x4b7ec 0xb9f1abb 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f3fd3 0xba38ebd 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:50.676 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:50.715 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x42198 0x3ecea 0xfc9b1 0x3870209 0xb85a732 0xb85a87a 0xb85a963 0xb85ac23 0xb85ad16 0xb85adf6 0xb85b076 0xb85b621 0xb85ba2c 0xb85bceb 0xba2da49 0xba312a0 0xb9e403b 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:50.715 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffac2a5d17 0xffffffffabfba036 0xffffffffabfbdce1 0xffffffffabfbee42 0xffffffffabfbf6b7 0xffffffffabfe4853 0xffffffffabf8e356 0xffffffffabf96a3b 0xffffffffabf96ebc 0xffffffffabc8c884 0xffffffffac8d1d62 0xffffffffaca00bb2 2024-09-11T14:38:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xba11a30 0xba295d2 0xba2979b 0xb9d1e3e 0xb9d64f1 0x984ae51 0x9b05eb6 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:50.830 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4aba4 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:50.830 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:50.887 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab61 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:50.888 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:50.920 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10798d 0xdd2e1 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:52.646 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:52.646 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-09-11T14:38:52.646 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-09-11T14:38:52.647 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-09-11T14:38:52.647 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:52.647 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-11T14:38:52.672 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T14:38:52.698 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:38:52.998 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:38:52.999 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:52,998 seastar - Reactor backend: linux-aio 2024-09-11T14:38:53.005 INFO:teuthology.orchestra.run.smithi059.stderr:==27996==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:38:53.114 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x156dc 0xdd312 0x4ab1c 0x45831ed 0x4614bc2 0x5fe3765 0x5fc04ab 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x3968bb0 0x396a38a 0x380ac18 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:53.114 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:53,134 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:38:53.135 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:53,134 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-11T14:38:53.136 INFO:teuthology.orchestra.run.smithi059.stderr:DEBUG 2024-09-11 14:38:53,135 [shard 0:main] seastar - reactor::drain 2024-09-11T14:38:53.665 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:53.665 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-09-11T14:38:53.665 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-09-11T14:38:53.666 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-09-11T14:38:53.666 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:53.666 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-11T14:38:53.720 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:38:53.724 INFO:teuthology.orchestra.run.smithi059.stderr:INFO 2024-09-11 14:38:53,724 seastar - Reactor backend: linux-aio 2024-09-11T14:38:53.732 INFO:teuthology.orchestra.run.smithi059.stderr:==27994==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:38:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133cca 0xdd332 0x4b14c 0x5ff0f1d 0x5ff11cf 0x5ff14a5 0x603fba3 0x603fda9 0x5fbfef6 0x5fc00d0 0x5fc065c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.032 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10bfb0 0x10a294 0xfe63e 0x5cca65c 0x5cca836 0x5ccaaed 0x5ccabe2 0xba24174 0xb9e408e 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:54.032 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.090 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x390ff 0x4ab1c 0x37a9f02 0xb0e1045 0xb0e152c 0xb0e168c 0xc4b9252 0xc4b9862 0xb0da5b9 0xb0d4c90 0xac9c741 0xad02411 0xaccefd1 0x994da0a 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:54.090 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.116 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4449e 0xfb4fb 0x376ee9b 0xba241a5 0xba2fac4 0xba2fcdf 0xba2fd8f 0xb9df75d 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:54.116 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.174 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xfc9df 0x3870209 0xba2d8b2 0xba312a0 0xb9e403b 0xb9ced56 0xb9cf13a 0x9ab9cb7 0x9b05946 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:54.174 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xc9206 0xc9e39 0x360e4e8 0x3b92070 0x3b92268 0xba052ce 0xba407e3 0xba40b6e 0xb9e0f68 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.269 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x39711fb 0x5c5c4d8 0xb9e1883 0xb9e3db8 0xb9ced56 0xb9cf13a 0xa35abab 0x9d382b4 0x9d370b2 0x994a08b 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:54.269 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.322 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab61 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:54.322 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.380 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x156cf 0xdd312 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:38:54.380 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:54.415 INFO:teuthology.orchestra.run.smithi059.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10798a 0xdd2e1 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:38:54.415 INFO:teuthology.orchestra.run.smithi059.stderr:kernel callstack: 2024-09-11T14:38:56.095 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:56.095 INFO:teuthology.orchestra.run.smithi059.stderr:Suppressions used: 2024-09-11T14:38:56.095 INFO:teuthology.orchestra.run.smithi059.stderr: count bytes template 2024-09-11T14:38:56.095 INFO:teuthology.orchestra.run.smithi059.stderr: 1 8 MallocExtension::Register 2024-09-11T14:38:56.095 INFO:teuthology.orchestra.run.smithi059.stderr:----------------------------------------------------- 2024-09-11T14:38:56.095 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-11T14:38:56.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T14:38:56.148 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T14:38:56.148 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:56.148 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T14:38:56.214 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:56.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T14:38:56.279 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:56.280 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T14:38:56.345 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:56.346 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T14:38:56.411 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:56.411 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T14:38:56.466 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T14:38:56.467 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T14:38:56.530 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBzq+FmSdxEFBAAt3xRrc1jLpQxRGb5zqdtRw== 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQB2q+FmxBHuNxAAhKpZpPauIaLEViFoHYATNw== 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQB6q+FmL/PsFhAACpyLbCl/naq3gKGuBlunqQ== 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQB9q+Fm36bbNBAAQEejip16aqdc7Gt5H7PA3w== 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-09-11T14:38:56.531 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBzq+FmzwqAFxAAmG0hBQv0tR+UGIVxIKH/hA== 2024-09-11T14:38:56.532 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T14:38:56.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:38:56.696 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:38:56.758 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:38:56.820 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T14:38:56.886 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T14:38:56.886 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T14:38:56.912 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T14:38:57.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:57.057+0000 7f1d0b112c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:57.057+0000 7f1d0b112c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T14:38:57.097 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:38:57.153 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T14:38:57.153 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T14:38:57.153 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T14:38:57.195 INFO:tasks.ceph.mon.a:Started 2024-09-11T14:38:57.195 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T14:38:57.196 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T14:38:57.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T14:38:57.198 INFO:tasks.ceph.mgr.x:Started 2024-09-11T14:38:57.198 DEBUG:tasks.ceph:set 0 configs 2024-09-11T14:38:57.198 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T14:38:57.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T14:38:57.281 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-09-11T14:38:57.280+0000 7f7afbe2ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.281 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-09-11T14:38:57.280+0000 7f7afbe2ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.281 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-09-11T14:38:57.280+0000 7f7afbe2ec00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.287 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-09-11T14:38:57.286+0000 7f0587839000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:57.306+0000 7f9f6e911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:57.318+0000 7f9f6e911640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.392 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-09-11T14:38:57.404 INFO:tasks.ceph:check_enable_crimson: True 2024-09-11T14:38:57.404 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-11T14:38:57.404 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-11T14:38:57.493 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-09-11T14:38:57.492+0000 7f0587839000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.494 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-09-11T14:38:57.492+0000 7f0587839000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:57.513+0000 7f286e0a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:57.522+0000 7f286e0a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:57.624 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-09-11T14:38:57.623+0000 7f7af27f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-11T14:38:57.624 INFO:tasks.ceph.mon.a.smithi059.stderr:Please be aware that this feature is experimental, untested, 2024-09-11T14:38:57.625 INFO:tasks.ceph.mon.a.smithi059.stderr:unsupported, and may result in data corruption, data loss, 2024-09-11T14:38:57.625 INFO:tasks.ceph.mon.a.smithi059.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-11T14:38:57.625 INFO:tasks.ceph.mon.a.smithi059.stderr:feature with important data. 2024-09-11T14:38:57.625 INFO:tasks.ceph.mon.a.smithi059.stderr: 2024-09-11T14:38:58.405 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T14:38:58.405 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:58.405 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T14:38:58.430 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:58.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T14:38:58.497 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:38:58.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T14:38:58.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 38039c61-9ba6-47ac-ada1-6058644a16a3 0 2024-09-11T14:38:58.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:58.707+0000 7f08b0591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:58.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:58.718+0000 7f08b0591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:58.800 INFO:teuthology.orchestra.run.smithi059.stdout:0 2024-09-11T14:38:58.813 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 6c1cd617-bba7-42b4-8e50-d725dae9bf2a 1 2024-09-11T14:38:58.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:58.916+0000 7f38918d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:58.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:58.927+0000 7f38918d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:59.011 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-09-11T14:38:59.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd new 3fa00c86-c849-4001-862e-9e53163090f0 2 2024-09-11T14:38:59.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:59.138+0000 7eff60823640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:59.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:59.149+0000 7eff60823640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:59.231 INFO:teuthology.orchestra.run.smithi059.stdout:2 2024-09-11T14:38:59.243 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-11T14:38:59.274 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T14:38:59.274 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-11T14:38:59.316 INFO:tasks.ceph.osd.0:Started 2024-09-11T14:38:59.316 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-11T14:38:59.348 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T14:38:59.348 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-11T14:38:59.391 INFO:tasks.ceph.osd.1:Started 2024-09-11T14:38:59.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-11T14:38:59.425 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T14:38:59.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-11T14:38:59.468 INFO:tasks.ceph.osd.2:Started 2024-09-11T14:38:59.468 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:38:59.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:59.591+0000 7fc31e311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:38:59.601+0000 7fc31e311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:38:59.701 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:38:59.701 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"fsid":"1af81b22-8a9d-4464-aa64-8b69c827c577","created":"2024-09-11T14:38:57.305066+0000","modified":"2024-09-11T14:38:59.230220+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:38:59.230220+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"38039c61-9ba6-47ac-ada1-6058644a16a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6c1cd617-bba7-42b4-8e50-d725dae9bf2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3fa00c86-c849-4001-862e-9e53163090f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:38:59.720 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T14:38:59.720 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T14:39:00.022 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:39:00.129 INFO:teuthology.misc.health.smithi059.stderr:2024-09-11T14:39:00.123+0000 7f4c7f591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:00.133 INFO:teuthology.misc.health.smithi059.stderr:2024-09-11T14:39:00.133+0000 7f4c7f591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:00.207 INFO:teuthology.misc.health.smithi059.stdout: 2024-09-11T14:39:00.207 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":6,"fsid":"1af81b22-8a9d-4464-aa64-8b69c827c577","created":"2024-09-11T14:38:57.305066+0000","modified":"2024-09-11T14:38:59.230220+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:38:59.230220+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"38039c61-9ba6-47ac-ada1-6058644a16a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6c1cd617-bba7-42b4-8e50-d725dae9bf2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3fa00c86-c849-4001-862e-9e53163090f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:39:00.218 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T14:39:03.248 INFO:tasks.ceph.mgr.x.smithi059.stderr:2024-09-11T14:39:03.247+0000 7f0524725640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T14:39:06.520 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:39:06.630 INFO:teuthology.misc.health.smithi059.stderr:2024-09-11T14:39:06.623+0000 7f4a8d26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:06.635 INFO:teuthology.misc.health.smithi059.stderr:2024-09-11T14:39:06.633+0000 7f4a8d26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:06.838 INFO:teuthology.misc.health.smithi059.stdout: 2024-09-11T14:39:06.839 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"1af81b22-8a9d-4464-aa64-8b69c827c577","created":"2024-09-11T14:38:57.305066+0000","modified":"2024-09-11T14:39:06.320059+0000","last_up_change":"2024-09-11T14:39:05.311247+0000","last_in_change":"2024-09-11T14:38:59.230220+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:39:06.266706+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38039c61-9ba6-47ac-ada1-6058644a16a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":823145234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":823145234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":823145234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":823145234}]},"public_addr":"172.21.15.59:6805/823145234","cluster_addr":"172.21.15.59:6804/823145234","heartbeat_back_addr":"172.21.15.59:6806/823145234","heartbeat_front_addr":"172.21.15.59:6809/823145234","state":["exists","up"]},{"osd":1,"uuid":"6c1cd617-bba7-42b4-8e50-d725dae9bf2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":1003425153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1003425153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1003425153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":1003425153}]},"public_addr":"172.21.15.59:6803/1003425153","cluster_addr":"172.21.15.59:6802/1003425153","heartbeat_back_addr":"172.21.15.59:6808/1003425153","heartbeat_front_addr":"172.21.15.59:6807/1003425153","state":["exists","up"]},{"osd":2,"uuid":"3fa00c86-c849-4001-862e-9e53163090f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":746975144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":746975144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":746975144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":746975144}]},"public_addr":"172.21.15.59:6812/746975144","cluster_addr":"172.21.15.59:6811/746975144","heartbeat_back_addr":"172.21.15.59:6813/746975144","heartbeat_front_addr":"172.21.15.59:6810/746975144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:39:06.852 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T14:39:06.852 INFO:tasks.ceph:Creating RBD pool 2024-09-11T14:39:06.852 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T14:39:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:06.962+0000 7f18a1191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:06.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:06.975+0000 7f18a1191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:07.325 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'rbd' created 2024-09-11T14:39:07.342 DEBUG:teuthology.orchestra.run.smithi059:> rbd --cluster ceph pool init rbd 2024-09-11T14:39:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:07.450+0000 7f2948674ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:07.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:07.451+0000 7f2948674ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:07.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:07.457+0000 7f2948674ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:10.626 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T14:39:10.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T14:39:10.627 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T14:39:10.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:10.735+0000 7f3ae6991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:10.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:10.744+0000 7f3ae6991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:10.944 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:39:10.957 INFO:teuthology.orchestra.run.smithi059.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-11T14:39:10.958 INFO:tasks.ceph_manager:config epoch is 1 2024-09-11T14:39:10.958 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T14:39:10.958 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T14:39:10.958 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T14:39:11.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:11.064+0000 7ff4b0791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:11.073+0000 7ff4b0791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:11.322 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:39:11.334 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1306069696},{"type":"v1","addr":"172.21.15.59:6801","nonce":1306069696}]},"active_addr":"172.21.15.59:6801/1306069696","active_change":"2024-09-11T14:39:02.231129+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":1573384188}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":2206758434}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":1522884492}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":2795514722}]}]} 2024-09-11T14:39:11.336 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T14:39:11.337 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T14:39:11.337 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:39:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:11.444+0000 7f62ce488640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:11.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:11.452+0000 7f62ce488640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:11.650 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:39:11.650 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":14,"fsid":"1af81b22-8a9d-4464-aa64-8b69c827c577","created":"2024-09-11T14:38:57.305066+0000","modified":"2024-09-11T14:39:10.538468+0000","last_up_change":"2024-09-11T14:39:05.311247+0000","last_in_change":"2024-09-11T14:38:59.230220+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:39:06.266706+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:39:07.179749+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38039c61-9ba6-47ac-ada1-6058644a16a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":823145234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":823145234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":823145234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":823145234}]},"public_addr":"172.21.15.59:6805/823145234","cluster_addr":"172.21.15.59:6804/823145234","heartbeat_back_addr":"172.21.15.59:6806/823145234","heartbeat_front_addr":"172.21.15.59:6809/823145234","state":["exists","up"]},{"osd":1,"uuid":"6c1cd617-bba7-42b4-8e50-d725dae9bf2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":1003425153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1003425153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1003425153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":1003425153}]},"public_addr":"172.21.15.59:6803/1003425153","cluster_addr":"172.21.15.59:6802/1003425153","heartbeat_back_addr":"172.21.15.59:6808/1003425153","heartbeat_front_addr":"172.21.15.59:6807/1003425153","state":["exists","up"]},{"osd":2,"uuid":"3fa00c86-c849-4001-862e-9e53163090f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":746975144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":746975144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":746975144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":746975144}]},"public_addr":"172.21.15.59:6812/746975144","cluster_addr":"172.21.15.59:6811/746975144","heartbeat_back_addr":"172.21.15.59:6813/746975144","heartbeat_front_addr":"172.21.15.59:6810/746975144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:39:11.663 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T14:39:11.663 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:39:11.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:11.768+0000 7f1ce8551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:11.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:11.777+0000 7f1ce8551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:11.975 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:39:11.975 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":14,"fsid":"1af81b22-8a9d-4464-aa64-8b69c827c577","created":"2024-09-11T14:38:57.305066+0000","modified":"2024-09-11T14:39:10.538468+0000","last_up_change":"2024-09-11T14:39:05.311247+0000","last_in_change":"2024-09-11T14:38:59.230220+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:39:06.266706+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:39:07.179749+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38039c61-9ba6-47ac-ada1-6058644a16a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":823145234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":823145234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":823145234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":823145234}]},"public_addr":"172.21.15.59:6805/823145234","cluster_addr":"172.21.15.59:6804/823145234","heartbeat_back_addr":"172.21.15.59:6806/823145234","heartbeat_front_addr":"172.21.15.59:6809/823145234","state":["exists","up"]},{"osd":1,"uuid":"6c1cd617-bba7-42b4-8e50-d725dae9bf2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":1003425153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1003425153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1003425153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":1003425153}]},"public_addr":"172.21.15.59:6803/1003425153","cluster_addr":"172.21.15.59:6802/1003425153","heartbeat_back_addr":"172.21.15.59:6808/1003425153","heartbeat_front_addr":"172.21.15.59:6807/1003425153","state":["exists","up"]},{"osd":2,"uuid":"3fa00c86-c849-4001-862e-9e53163090f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":746975144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":746975144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":746975144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":746975144}]},"public_addr":"172.21.15.59:6812/746975144","cluster_addr":"172.21.15.59:6811/746975144","heartbeat_back_addr":"172.21.15.59:6813/746975144","heartbeat_front_addr":"172.21.15.59:6810/746975144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:39:11.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T14:39:11.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T14:39:11.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T14:39:12.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.102+0000 7f1e57cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.102+0000 7f27cf8d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.102+0000 7efe3c09c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.112+0000 7f27cf8d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.113+0000 7efe3c09c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.113+0000 7f1e57cc8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.149 INFO:teuthology.orchestra.run.smithi059.stdout:34359738375 2024-09-11T14:39:12.149 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:39:12.150 INFO:teuthology.orchestra.run.smithi059.stdout:38654705670 2024-09-11T14:39:12.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:39:12.151 INFO:teuthology.orchestra.run.smithi059.stdout:34359738375 2024-09-11T14:39:12.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:39:12.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.258+0000 7f0fd1711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.269+0000 7f0fd1711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.304+0000 7f023d246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.307+0000 7f6848791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.314+0000 7f023d246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:12.317+0000 7f6848791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:12.482 INFO:teuthology.orchestra.run.smithi059.stdout:34359738373 2024-09-11T14:39:12.496 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738373 for osd.1 2024-09-11T14:39:12.523 INFO:teuthology.orchestra.run.smithi059.stdout:38654705668 2024-09-11T14:39:12.529 INFO:teuthology.orchestra.run.smithi059.stdout:34359738372 2024-09-11T14:39:12.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705668 for osd.2 2024-09-11T14:39:12.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738372 for osd.0 2024-09-11T14:39:13.497 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:39:13.540 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:39:13.543 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:39:13.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:13.624+0000 7f94fc791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:13.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:13.639+0000 7f94fc791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:13.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:13.662+0000 7f297bb51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:13.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:13.662+0000 7fcefdc8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:13.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:13.672+0000 7f297bb51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:13.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:13.672+0000 7fcefdc8e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:13.854 INFO:teuthology.orchestra.run.smithi059.stdout:34359738375 2024-09-11T14:39:13.868 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2024-09-11T14:39:13.868 DEBUG:teuthology.parallel:result is None 2024-09-11T14:39:13.880 INFO:teuthology.orchestra.run.smithi059.stdout:34359738375 2024-09-11T14:39:13.887 INFO:teuthology.orchestra.run.smithi059.stdout:38654705670 2024-09-11T14:39:13.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.0 2024-09-11T14:39:13.895 DEBUG:teuthology.parallel:result is None 2024-09-11T14:39:13.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2024-09-11T14:39:13.900 DEBUG:teuthology.parallel:result is None 2024-09-11T14:39:13.901 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:39:13.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:39:14.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:14.047+0000 7f26d0151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:14.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:14.055+0000 7f26d0151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:14.251 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:39:14.252 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-11T14:39:14.264 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-11T14:39:14.239471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3348,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889836,"statfs":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.844868"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.581702+0000","last_change":"2024-09-11T14:39:10.591513+0000","last_active":"2024-09-11T14:39:10.581702+0000","last_peered":"2024-09-11T14:39:10.581702+0000","last_clean":"2024-09-11T14:39:10.581702+0000","last_became_active":"2024-09-11T14:39:08.649048+0000","last_became_peered":"2024-09-11T14:39:08.649048+0000","last_unstale":"2024-09-11T14:39:10.581702+0000","last_undegraded":"2024-09-11T14:39:10.581702+0000","last_fullsized":"2024-09-11T14:39:10.581702+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.577471+0000","last_change":"2024-09-11T14:39:10.590719+0000","last_active":"2024-09-11T14:39:10.577471+0000","last_peered":"2024-09-11T14:39:10.577471+0000","last_clean":"2024-09-11T14:39:10.577471+0000","last_became_active":"2024-09-11T14:39:08.671767+0000","last_became_peered":"2024-09-11T14:39:08.671767+0000","last_unstale":"2024-09-11T14:39:10.577471+0000","last_undegraded":"2024-09-11T14:39:10.577471+0000","last_fullsized":"2024-09-11T14:39:10.577471+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.587665+0000","last_change":"2024-09-11T14:39:10.595741+0000","last_active":"2024-09-11T14:39:10.587665+0000","last_peered":"2024-09-11T14:39:10.587665+0000","last_clean":"2024-09-11T14:39:10.587665+0000","last_became_active":"2024-09-11T14:39:08.728005+0000","last_became_peered":"2024-09-11T14:39:08.728005+0000","last_unstale":"2024-09-11T14:39:10.587665+0000","last_undegraded":"2024-09-11T14:39:10.587665+0000","last_fullsized":"2024-09-11T14:39:10.587665+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.582548+0000","last_change":"2024-09-11T14:39:10.592063+0000","last_active":"2024-09-11T14:39:10.582548+0000","last_peered":"2024-09-11T14:39:10.582548+0000","last_clean":"2024-09-11T14:39:10.582548+0000","last_became_active":"2024-09-11T14:39:08.660946+0000","last_became_peered":"2024-09-11T14:39:08.660946+0000","last_unstale":"2024-09-11T14:39:10.582548+0000","last_undegraded":"2024-09-11T14:39:10.582548+0000","last_fullsized":"2024-09-11T14:39:10.582548+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.598829+0000","last_change":"2024-09-11T14:39:10.598829+0000","last_active":"2024-09-11T14:39:10.598829+0000","last_peered":"2024-09-11T14:39:10.598829+0000","last_clean":"2024-09-11T14:39:10.598829+0000","last_became_active":"2024-09-11T14:39:08.696312+0000","last_became_peered":"2024-09-11T14:39:08.696312+0000","last_unstale":"2024-09-11T14:39:10.598829+0000","last_undegraded":"2024-09-11T14:39:10.598829+0000","last_fullsized":"2024-09-11T14:39:10.598829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.589987+0000","last_change":"2024-09-11T14:39:10.595170+0000","last_active":"2024-09-11T14:39:10.589987+0000","last_peered":"2024-09-11T14:39:10.589987+0000","last_clean":"2024-09-11T14:39:10.589987+0000","last_became_active":"2024-09-11T14:39:08.722304+0000","last_became_peered":"2024-09-11T14:39:08.722304+0000","last_unstale":"2024-09-11T14:39:10.589987+0000","last_undegraded":"2024-09-11T14:39:10.589987+0000","last_fullsized":"2024-09-11T14:39:10.589987+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.583612+0000","last_change":"2024-09-11T14:39:10.588768+0000","last_active":"2024-09-11T14:39:10.583612+0000","last_peered":"2024-09-11T14:39:10.583612+0000","last_clean":"2024-09-11T14:39:10.583612+0000","last_became_active":"2024-09-11T14:39:08.670002+0000","last_became_peered":"2024-09-11T14:39:08.670002+0000","last_unstale":"2024-09-11T14:39:10.583612+0000","last_undegraded":"2024-09-11T14:39:10.583612+0000","last_fullsized":"2024-09-11T14:39:10.583612+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.588541+0000","last_change":"2024-09-11T14:39:10.596036+0000","last_active":"2024-09-11T14:39:10.588541+0000","last_peered":"2024-09-11T14:39:10.588541+0000","last_clean":"2024-09-11T14:39:10.588541+0000","last_became_active":"2024-09-11T14:39:08.669437+0000","last_became_peered":"2024-09-11T14:39:08.669437+0000","last_unstale":"2024-09-11T14:39:10.588541+0000","last_undegraded":"2024-09-11T14:39:10.588541+0000","last_fullsized":"2024-09-11T14:39:10.588541+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.578120+0000","last_change":"2024-09-11T14:39:07.488096+0000","last_active":"2024-09-11T14:39:10.578120+0000","last_peered":"2024-09-11T14:39:10.578120+0000","last_clean":"2024-09-11T14:39:10.578120+0000","last_became_active":"2024-09-11T14:39:07.454105+0000","last_became_peered":"2024-09-11T14:39:07.454105+0000","last_unstale":"2024-09-11T14:39:10.578120+0000","last_undegraded":"2024-09-11T14:39:10.578120+0000","last_fullsized":"2024-09-11T14:39:10.578120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_clean_scrub_stamp":"2024-09-11T14:39:06.320059+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":428,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297300,"statfs":{"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1588,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296140,"statfs":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296396,"statfs":{"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:39:14.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:39:14.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:14.372+0000 7fd497148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:14.380+0000 7fd497148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:14.581 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:39:14.582 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-11T14:39:14.593 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-11T14:39:14.239471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3348,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889836,"statfs":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.844868"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.581702+0000","last_change":"2024-09-11T14:39:10.591513+0000","last_active":"2024-09-11T14:39:10.581702+0000","last_peered":"2024-09-11T14:39:10.581702+0000","last_clean":"2024-09-11T14:39:10.581702+0000","last_became_active":"2024-09-11T14:39:08.649048+0000","last_became_peered":"2024-09-11T14:39:08.649048+0000","last_unstale":"2024-09-11T14:39:10.581702+0000","last_undegraded":"2024-09-11T14:39:10.581702+0000","last_fullsized":"2024-09-11T14:39:10.581702+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.577471+0000","last_change":"2024-09-11T14:39:10.590719+0000","last_active":"2024-09-11T14:39:10.577471+0000","last_peered":"2024-09-11T14:39:10.577471+0000","last_clean":"2024-09-11T14:39:10.577471+0000","last_became_active":"2024-09-11T14:39:08.671767+0000","last_became_peered":"2024-09-11T14:39:08.671767+0000","last_unstale":"2024-09-11T14:39:10.577471+0000","last_undegraded":"2024-09-11T14:39:10.577471+0000","last_fullsized":"2024-09-11T14:39:10.577471+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.587665+0000","last_change":"2024-09-11T14:39:10.595741+0000","last_active":"2024-09-11T14:39:10.587665+0000","last_peered":"2024-09-11T14:39:10.587665+0000","last_clean":"2024-09-11T14:39:10.587665+0000","last_became_active":"2024-09-11T14:39:08.728005+0000","last_became_peered":"2024-09-11T14:39:08.728005+0000","last_unstale":"2024-09-11T14:39:10.587665+0000","last_undegraded":"2024-09-11T14:39:10.587665+0000","last_fullsized":"2024-09-11T14:39:10.587665+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.582548+0000","last_change":"2024-09-11T14:39:10.592063+0000","last_active":"2024-09-11T14:39:10.582548+0000","last_peered":"2024-09-11T14:39:10.582548+0000","last_clean":"2024-09-11T14:39:10.582548+0000","last_became_active":"2024-09-11T14:39:08.660946+0000","last_became_peered":"2024-09-11T14:39:08.660946+0000","last_unstale":"2024-09-11T14:39:10.582548+0000","last_undegraded":"2024-09-11T14:39:10.582548+0000","last_fullsized":"2024-09-11T14:39:10.582548+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.598829+0000","last_change":"2024-09-11T14:39:10.598829+0000","last_active":"2024-09-11T14:39:10.598829+0000","last_peered":"2024-09-11T14:39:10.598829+0000","last_clean":"2024-09-11T14:39:10.598829+0000","last_became_active":"2024-09-11T14:39:08.696312+0000","last_became_peered":"2024-09-11T14:39:08.696312+0000","last_unstale":"2024-09-11T14:39:10.598829+0000","last_undegraded":"2024-09-11T14:39:10.598829+0000","last_fullsized":"2024-09-11T14:39:10.598829+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.589987+0000","last_change":"2024-09-11T14:39:10.595170+0000","last_active":"2024-09-11T14:39:10.589987+0000","last_peered":"2024-09-11T14:39:10.589987+0000","last_clean":"2024-09-11T14:39:10.589987+0000","last_became_active":"2024-09-11T14:39:08.722304+0000","last_became_peered":"2024-09-11T14:39:08.722304+0000","last_unstale":"2024-09-11T14:39:10.589987+0000","last_undegraded":"2024-09-11T14:39:10.589987+0000","last_fullsized":"2024-09-11T14:39:10.589987+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.583612+0000","last_change":"2024-09-11T14:39:10.588768+0000","last_active":"2024-09-11T14:39:10.583612+0000","last_peered":"2024-09-11T14:39:10.583612+0000","last_clean":"2024-09-11T14:39:10.583612+0000","last_became_active":"2024-09-11T14:39:08.670002+0000","last_became_peered":"2024-09-11T14:39:08.670002+0000","last_unstale":"2024-09-11T14:39:10.583612+0000","last_undegraded":"2024-09-11T14:39:10.583612+0000","last_fullsized":"2024-09-11T14:39:10.583612+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.588541+0000","last_change":"2024-09-11T14:39:10.596036+0000","last_active":"2024-09-11T14:39:10.588541+0000","last_peered":"2024-09-11T14:39:10.588541+0000","last_clean":"2024-09-11T14:39:10.588541+0000","last_became_active":"2024-09-11T14:39:08.669437+0000","last_became_peered":"2024-09-11T14:39:08.669437+0000","last_unstale":"2024-09-11T14:39:10.588541+0000","last_undegraded":"2024-09-11T14:39:10.588541+0000","last_fullsized":"2024-09-11T14:39:10.588541+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-11T14:39:10.578120+0000","last_change":"2024-09-11T14:39:07.488096+0000","last_active":"2024-09-11T14:39:10.578120+0000","last_peered":"2024-09-11T14:39:10.578120+0000","last_clean":"2024-09-11T14:39:10.578120+0000","last_became_active":"2024-09-11T14:39:07.454105+0000","last_became_peered":"2024-09-11T14:39:07.454105+0000","last_unstale":"2024-09-11T14:39:10.578120+0000","last_undegraded":"2024-09-11T14:39:10.578120+0000","last_fullsized":"2024-09-11T14:39:10.578120+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_clean_scrub_stamp":"2024-09-11T14:39:06.320059+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":428,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297300,"statfs":{"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1588,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296140,"statfs":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296396,"statfs":{"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:39:14.594 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:39:14.594 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T14:39:14.594 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T14:39:14.594 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T14:39:14.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:14.700+0000 7f1815d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:14.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T14:39:14.707+0000 7f1815d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:39:14.942 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T14:39:14.942 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T14:39:14.954 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T14:39:14.955 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T14:39:14.969 INFO:tasks.workunit:Pulling workunits from ref dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 2024-09-11T14:39:14.969 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T14:39:14.969 DEBUG:teuthology.orchestra.run.smithi059:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:39:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:39:14.985 INFO:teuthology.orchestra.run.smithi059.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T14:39:14.986 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:39:15.042 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-11T14:39:15.042 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-11T14:39:15.098 INFO:tasks.workunit:timeout=3h 2024-09-11T14:39:15.098 INFO:tasks.workunit:cleanup=True 2024-09-11T14:39:15.098 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/matan-b/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 2024-09-11T14:39:15.154 INFO:tasks.workunit.client.0.smithi059.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T14:40:00.082 INFO:tasks.workunit.client.0.smithi059.stderr:Note: switching to 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'. 2024-09-11T14:40:00.082 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-09-11T14:40:00.082 INFO:tasks.workunit.client.0.smithi059.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr: git switch -c 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr:Or undo this operation with: 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr: git switch - 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-09-11T14:40:00.083 INFO:tasks.workunit.client.0.smithi059.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T14:40:00.084 INFO:tasks.workunit.client.0.smithi059.stderr: 2024-09-11T14:40:00.084 INFO:tasks.workunit.client.0.smithi059.stderr:HEAD is now at dbebf1a3db6 qa/config/seastore: enable all subsystem debug 2024-09-11T14:40:00.086 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T14:40:00.151 INFO:tasks.workunit.client.0.smithi059.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T14:40:00.153 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T14:40:00.153 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T14:40:00.774 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T14:40:00.848 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T14:40:00.890 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T14:40:00.892 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T14:40:00.892 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T14:40:00.933 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T14:40:00.936 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-09-11T14:40:00.936 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T14:40:00.993 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2024-09-11T14:40:00.995 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2024-09-11T14:40:00.996 DEBUG:teuthology.orchestra.run.smithi059:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-09-11T14:40:01.103 INFO:tasks.workunit.client.0.smithi059.stdout:seed 29996 2024-09-11T14:40:01.104 INFO:tasks.workunit.client.0.smithi059.stdout:Note: Google Test filter = TestLibRBD.* 2024-09-11T14:40:01.104 INFO:tasks.workunit.client.0.smithi059.stdout:[==========] Running 171 tests from 1 test suite. 2024-09-11T14:40:01.104 INFO:tasks.workunit.client.0.smithi059.stdout:[----------] Global test environment set-up. 2024-09-11T14:40:01.104 INFO:tasks.workunit.client.0.smithi059.stdout:[----------] 171 tests from TestLibRBD 2024-09-11T14:40:01.155 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-09-11T14:40:02.346 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:04.702 INFO:tasks.workunit.client.0.smithi059.stdout:image has size 2097152 and order 22 2024-09-11T14:40:04.702 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.CreateAndStat (3547 ms) 2024-09-11T14:40:04.702 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-09-11T14:40:04.702 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:04.950 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (249 ms) 2024-09-11T14:40:04.950 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-09-11T14:40:04.951 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:05.201 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.CreateAndStatPP (250 ms) 2024-09-11T14:40:05.201 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.GetId 2024-09-11T14:40:05.201 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:05.570 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.GetId (369 ms) 2024-09-11T14:40:05.570 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-09-11T14:40:05.570 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:05.935 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.GetIdPP (366 ms) 2024-09-11T14:40:05.935 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-09-11T14:40:05.936 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:06.187 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (251 ms) 2024-09-11T14:40:06.187 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-09-11T14:40:06.187 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:06.436 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (249 ms) 2024-09-11T14:40:06.436 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-09-11T14:40:06.436 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:06.690 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (255 ms) 2024-09-11T14:40:06.691 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-09-11T14:40:06.691 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:06.927 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (236 ms) 2024-09-11T14:40:06.927 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.OpenAio 2024-09-11T14:40:06.927 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:07.164 INFO:tasks.workunit.client.0.smithi059.stdout:image has size 2097152 and order 22 2024-09-11T14:40:07.164 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.OpenAio (237 ms) 2024-09-11T14:40:07.164 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-09-11T14:40:07.178 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.OpenAioFail (14 ms) 2024-09-11T14:40:07.178 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-09-11T14:40:07.178 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:07.564 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.OpenAioPP (386 ms) 2024-09-11T14:40:07.564 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-09-11T14:40:07.577 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.OpenAioFailPP (14 ms) 2024-09-11T14:40:07.577 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-09-11T14:40:07.577 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:08.272 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ResizeAndStat (694 ms) 2024-09-11T14:40:08.272 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-09-11T14:40:08.272 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:08.827 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (556 ms) 2024-09-11T14:40:08.828 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-09-11T14:40:08.828 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:09.462 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (635 ms) 2024-09-11T14:40:09.462 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-09-11T14:40:09.462 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:10.105 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (642 ms) 2024-09-11T14:40:10.105 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-09-11T14:40:12.098 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:13.614 INFO:tasks.workunit.client.0.smithi059.stdout:image: image18 2024-09-11T14:40:13.614 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image18 2024-09-11T14:40:13.614 INFO:tasks.workunit.client.0.smithi059.stdout:found image18 2024-09-11T14:40:13.614 INFO:tasks.workunit.client.0.smithi059.stdout:erased image18 2024-09-11T14:40:13.614 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:image: image18 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:image: image19 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image18 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:found image18 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:erased image18 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image19 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:found image19 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:erased image19 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:image: image19 2024-09-11T14:40:14.825 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image19 2024-09-11T14:40:14.826 INFO:tasks.workunit.client.0.smithi059.stdout:found image19 2024-09-11T14:40:14.826 INFO:tasks.workunit.client.0.smithi059.stdout:erased image19 2024-09-11T14:40:14.826 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (4720 ms) 2024-09-11T14:40:14.826 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-09-11T14:40:16.397 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:18.676 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 1 2024-09-11T14:40:18.676 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:40:18.676 INFO:tasks.workunit.client.0.smithi059.stdout:image: image20 2024-09-11T14:40:18.677 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image20 2024-09-11T14:40:18.677 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:18.821 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 2 2024-09-11T14:40:18.821 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:40:18.821 INFO:tasks.workunit.client.0.smithi059.stdout:image: image20 2024-09-11T14:40:18.821 INFO:tasks.workunit.client.0.smithi059.stdout:image: image21 2024-09-11T14:40:18.821 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image20 2024-09-11T14:40:18.821 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image21 2024-09-11T14:40:19.844 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 1 2024-09-11T14:40:19.844 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:40:19.845 INFO:tasks.workunit.client.0.smithi059.stdout:image: image21 2024-09-11T14:40:19.845 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image21 2024-09-11T14:40:19.845 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (5020 ms) 2024-09-11T14:40:19.845 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCopy 2024-09-11T14:40:21.414 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:31.534 INFO:tasks.workunit.client.0.smithi059.stdout:image: image22 2024-09-11T14:40:31.534 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image22 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:found image22 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:erased image22 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:image: image22 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:image: image23 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image22 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:found image22 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:erased image22 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image23 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:found image23 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:erased image23 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:100.00% done 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:image: image22 2024-09-11T14:40:31.535 INFO:tasks.workunit.client.0.smithi059.stdout:image: image23 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:image: image24 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image22 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:found image22 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:erased image22 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image23 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:found image23 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:erased image23 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image24 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:found image24 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:erased image24 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCopy (11689 ms) 2024-09-11T14:40:31.536 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-09-11T14:40:33.452 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:40:41.587 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 1 2024-09-11T14:40:41.587 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:40:41.587 INFO:tasks.workunit.client.0.smithi059.stdout:image: image25 2024-09-11T14:40:41.587 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image25 2024-09-11T14:40:41.937 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 2 2024-09-11T14:40:41.937 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:40:41.937 INFO:tasks.workunit.client.0.smithi059.stdout:image: image25 2024-09-11T14:40:41.937 INFO:tasks.workunit.client.0.smithi059.stdout:image: image26 2024-09-11T14:40:41.937 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image25 2024-09-11T14:40:41.937 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image26 2024-09-11T14:40:42.889 INFO:tasks.workunit.client.0.smithi059.stdout:100.00% done 2024-09-11T14:40:42.889 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 3 2024-09-11T14:40:42.889 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 3 2024-09-11T14:40:42.889 INFO:tasks.workunit.client.0.smithi059.stdout:image: image25 2024-09-11T14:40:42.889 INFO:tasks.workunit.client.0.smithi059.stdout:image: image26 2024-09-11T14:40:42.889 INFO:tasks.workunit.client.0.smithi059.stdout:image: image27 2024-09-11T14:40:42.889 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image25 2024-09-11T14:40:42.889 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image26 2024-09-11T14:40:42.890 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image27 2024-09-11T14:40:43.603 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCopyPP (12070 ms) 2024-09-11T14:40:43.604 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-09-11T14:40:45.479 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:04.880 INFO:tasks.workunit.client.0.smithi059.stdout:image: image28 2024-09-11T14:41:04.880 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image28 2024-09-11T14:41:04.880 INFO:tasks.workunit.client.0.smithi059.stdout:found image28 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:erased image28 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:image: image28 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:image: image29 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image28 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:found image28 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:erased image28 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image29 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:found image29 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:erased image29 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:100.00% done 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:image: image28 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:image: image29 2024-09-11T14:41:04.881 INFO:tasks.workunit.client.0.smithi059.stdout:image: image30 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image28 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:found image28 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:erased image28 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image29 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:found image29 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:erased image29 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image30 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:found image30 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:erased image30 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:image: image28 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:image: image29 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:image: image30 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:image: image31 2024-09-11T14:41:04.882 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image28 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:found image28 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:erased image28 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image29 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:found image29 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:erased image29 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image30 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:found image30 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:erased image30 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image31 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:found image31 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:erased image31 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:image: image28 2024-09-11T14:41:04.883 INFO:tasks.workunit.client.0.smithi059.stdout:image: image29 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:image: image30 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:image: image31 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:image: image32 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image28 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:found image28 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:erased image28 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image29 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:found image29 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:erased image29 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image30 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:found image30 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:erased image30 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image31 2024-09-11T14:41:04.884 INFO:tasks.workunit.client.0.smithi059.stdout:found image31 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:erased image31 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image32 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:found image32 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:erased image32 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:100.00% done 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:image: image28 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:image: image29 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:image: image30 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:image: image31 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:image: image32 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:image: image33 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image28 2024-09-11T14:41:04.885 INFO:tasks.workunit.client.0.smithi059.stdout:found image28 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:erased image28 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image29 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:found image29 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:erased image29 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image30 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:found image30 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:erased image30 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image31 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:found image31 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:erased image31 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image32 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:found image32 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:erased image32 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image33 2024-09-11T14:41:04.886 INFO:tasks.workunit.client.0.smithi059.stdout:found image33 2024-09-11T14:41:04.887 INFO:tasks.workunit.client.0.smithi059.stdout:erased image33 2024-09-11T14:41:04.887 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestDeepCopy (21276 ms) 2024-09-11T14:41:04.887 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-09-11T14:41:06.528 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:15.720 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 1 2024-09-11T14:41:15.720 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:41:15.720 INFO:tasks.workunit.client.0.smithi059.stdout:image: image34 2024-09-11T14:41:15.721 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image34 2024-09-11T14:41:16.220 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 2 2024-09-11T14:41:16.221 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:41:16.221 INFO:tasks.workunit.client.0.smithi059.stdout:image: image34 2024-09-11T14:41:16.221 INFO:tasks.workunit.client.0.smithi059.stdout:image: image35 2024-09-11T14:41:16.221 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image34 2024-09-11T14:41:16.221 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image35 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:100.00% done 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:num images is: 3 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 3 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:image: image34 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:image: image35 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:image: image36 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image34 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image35 2024-09-11T14:41:17.337 INFO:tasks.workunit.client.0.smithi059.stdout:expected = image36 2024-09-11T14:41:18.088 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (13208 ms) 2024-09-11T14:41:18.088 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-09-11T14:41:18.088 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:22.707 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 1 2024-09-11T14:41:22.707 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:41:22.707 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap1 2024-09-11T14:41:22.707 INFO:tasks.workunit.client.0.smithi059.stdout:found snap1 with size 2097152 2024-09-11T14:41:22.707 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 2 2024-09-11T14:41:22.707 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap1 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap2 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:found snap1 with size 2097152 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:found snap2 with size 4194304 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 1 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap2 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:found snap2 with size 4194304 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 0 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (4619 ms) 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-09-11T14:41:22.708 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:24.788 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (2082 ms) 2024-09-11T14:41:24.788 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-09-11T14:41:24.788 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:25.691 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 1 2024-09-11T14:41:25.691 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:41:25.691 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap1 2024-09-11T14:41:25.691 INFO:tasks.workunit.client.0.smithi059.stdout:found snap1 with size 2097152 2024-09-11T14:41:26.706 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 2 2024-09-11T14:41:26.706 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:41:26.706 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap1 2024-09-11T14:41:26.706 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap2 2024-09-11T14:41:26.706 INFO:tasks.workunit.client.0.smithi059.stdout:found snap1 with size 2097152 2024-09-11T14:41:26.706 INFO:tasks.workunit.client.0.smithi059.stdout:found snap2 with size 4194304 2024-09-11T14:41:27.683 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 1 2024-09-11T14:41:27.683 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:41:27.683 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap2 2024-09-11T14:41:27.683 INFO:tasks.workunit.client.0.smithi059.stdout:found snap2 with size 4194304 2024-09-11T14:41:28.702 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 0 2024-09-11T14:41:28.702 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T14:41:28.774 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (3986 ms) 2024-09-11T14:41:28.775 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-09-11T14:41:28.775 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:34.701 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 0 2024-09-11T14:41:34.701 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T14:41:34.766 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (5991 ms) 2024-09-11T14:41:34.766 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-09-11T14:41:34.766 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:35.712 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 1 2024-09-11T14:41:35.712 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:41:35.712 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap1 2024-09-11T14:41:35.712 INFO:tasks.workunit.client.0.smithi059.stdout:found snap1 with size 2097152 2024-09-11T14:41:36.814 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 2 2024-09-11T14:41:36.815 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:41:36.815 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap1 2024-09-11T14:41:36.815 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap2 2024-09-11T14:41:36.815 INFO:tasks.workunit.client.0.smithi059.stdout:found snap1 with size 2097152 2024-09-11T14:41:36.815 INFO:tasks.workunit.client.0.smithi059.stdout:found snap2 with size 4194304 2024-09-11T14:41:36.924 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 2 2024-09-11T14:41:36.925 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:41:36.925 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap1-rename 2024-09-11T14:41:36.925 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap2 2024-09-11T14:41:36.925 INFO:tasks.workunit.client.0.smithi059.stdout:found snap1-rename with size 2097152 2024-09-11T14:41:36.925 INFO:tasks.workunit.client.0.smithi059.stdout:found snap2 with size 4194304 2024-09-11T14:41:37.814 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 1 2024-09-11T14:41:37.814 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:41:37.814 INFO:tasks.workunit.client.0.smithi059.stdout:snap: snap2-rename 2024-09-11T14:41:37.814 INFO:tasks.workunit.client.0.smithi059.stdout:found snap2-rename with size 4194304 2024-09-11T14:41:38.742 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 0 2024-09-11T14:41:38.742 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T14:41:38.822 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (4056 ms) 2024-09-11T14:41:38.822 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-09-11T14:41:40.443 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:40.443 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:40.443 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:40.443 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:41:47.077 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (8255 ms) 2024-09-11T14:41:47.077 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-09-11T14:41:51.650 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:42:09.939 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.RemoveFullTry (22862 ms) 2024-09-11T14:42:09.939 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-09-11T14:42:16.725 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:42:38.349 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:42:39.766 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:42:40.918 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:42:42.086 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:42:43.222 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:42:45.021 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.RemoveFullTryDataPool (35082 ms) 2024-09-11T14:42:45.021 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-09-11T14:42:47.793 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:43:09.803 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespace (24782 ms) 2024-09-11T14:43:09.803 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-09-11T14:43:14.488 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:43:38.382 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:43:39.888 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:43:41.038 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:43:42.262 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:43:44.475 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespaceDataPool (34673 ms) 2024-09-11T14:43:44.475 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestIO 2024-09-11T14:43:44.475 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:44:19.133 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:44:19.133 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.134 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.135 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:44:19.136 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.137 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.138 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:44:19.139 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:discard: 512~512 = 512 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:aio discard: 1536~512 = 0 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.140 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.141 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.142 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.143 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.144 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.145 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.146 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.147 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.148 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:19.149 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:19.149 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.149 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:19.149 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.149 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.149 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:19.149 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.150 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.150 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:19.150 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:19.150 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:19.150 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:19.150 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:19.150 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:19.151 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:19.151 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.492 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.492 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.493 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.494 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.495 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.496 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.497 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.498 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.499 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:44:48.500 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestIO (64016 ms) 2024-09-11T14:44:48.501 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-09-11T14:44:48.502 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:45:33.929 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:45:33.929 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:45:33.929 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:45:33.929 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:45:33.929 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:45:33.929 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.929 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:45:33.929 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.930 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:45:33.931 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:45:33.932 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.933 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:45:33.934 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:45:33.935 INFO:tasks.workunit.client.0.smithi059.stdout:discard: 512~512 = 512 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:aio discard: 1536~512 = 0 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.936 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:45:33.937 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.938 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.939 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.940 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.941 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.942 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.943 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:45:33.944 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.673 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.673 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.673 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.673 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.673 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.673 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:46:14.674 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.675 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.676 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.677 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.678 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.679 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.680 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.681 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 4 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (86181 ms) 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-09-11T14:46:14.682 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:46:23.419 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:46:23.419 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:46:23.419 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:46:23.419 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:46:23.419 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:46:23.420 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:46:23.421 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: -84 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:495: Failure 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout:Expected equality of these values: 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout: len 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout: Which is: 512 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout: static_cast(written) 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout: Which is: 18446744073709551532 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:630: Failure 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout:Value of: passed 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout: Actual: false 2024-09-11T14:46:23.422 INFO:tasks.workunit.client.0.smithi059.stdout:Expected: true 2024-09-11T14:46:29.260 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 4 2024-09-11T14:46:29.260 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:46:29.260 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:46:29.260 INFO:tasks.workunit.client.0.smithi059.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 (14587 ms) 2024-09-11T14:46:29.260 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-09-11T14:46:29.260 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:48:03.382 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 4 2024-09-11T14:48:03.382 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 4 2024-09-11T14:48:03.382 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 4 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:read: 4 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCloneEncryption (94122 ms) 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-09-11T14:48:03.383 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:48:22.651 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (19269 ms) 2024-09-11T14:48:22.651 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-09-11T14:48:22.651 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:48:42.212 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (19561 ms) 2024-09-11T14:48:42.212 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-09-11T14:48:42.212 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:48:47.467 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (5256 ms) 2024-09-11T14:48:47.468 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-09-11T14:48:47.468 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:48:54.387 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (6919 ms) 2024-09-11T14:48:54.387 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-11T14:48:54.388 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:2905: Skipped 2024-09-11T14:48:54.388 INFO:tasks.workunit.client.0.smithi059.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:48:54.388 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2024-09-11T14:48:54.388 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-09-11T14:48:54.388 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:51:25.411 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.EncryptionLoadFormatMismatch (151024 ms) 2024-09-11T14:51:25.412 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-09-11T14:51:25.412 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:51:40.399 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.EncryptedResize (14988 ms) 2024-09-11T14:51:40.400 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-09-11T14:51:40.400 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:51:57.355 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.EncryptedFlattenSmallData (16956 ms) 2024-09-11T14:51:57.355 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-09-11T14:51:57.355 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:52:31.017 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.018 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:52:31.019 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:52:31.020 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:52:31.021 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.022 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:52:31.023 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:52:31.024 INFO:tasks.workunit.client.0.smithi059.stdout:discard: 512~512 = 512 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:aio discard: 1536~512 = 0 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.025 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:52:31.026 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.027 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.028 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.029 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.030 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.031 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.032 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.033 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:31.034 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.770 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.771 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.771 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.771 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.771 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.771 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.771 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.771 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:59.772 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.773 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.773 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:59.773 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.773 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.773 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.773 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.773 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.773 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.774 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.775 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.776 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:59.776 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.776 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.776 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.776 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.776 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.776 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.776 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.777 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.778 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.779 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.780 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (62415 ms) 2024-09-11T14:52:59.781 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-09-11T14:53:00.861 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:04.050 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:04.050 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:04.050 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:04.050 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.051 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:04.052 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:04.053 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:discard: 512~512 = 512 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:04.054 INFO:tasks.workunit.client.0.smithi059.stdout:aio discard: 1536~512 = 0 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestDataPoolIO (4280 ms) 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-09-11T14:53:04.055 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:04.528 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (477 ms) 2024-09-11T14:53:04.528 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-09-11T14:53:04.528 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:05.038 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (510 ms) 2024-09-11T14:53:05.038 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-09-11T14:53:05.038 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:05.569 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (532 ms) 2024-09-11T14:53:05.569 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-09-11T14:53:05.569 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:06.068 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (498 ms) 2024-09-11T14:53:06.068 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-09-11T14:53:06.068 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:07.149 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (1081 ms) 2024-09-11T14:53:07.149 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-09-11T14:53:07.149 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:08.673 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (1525 ms) 2024-09-11T14:53:08.673 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-09-11T14:53:08.673 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:09.942 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (1268 ms) 2024-09-11T14:53:09.942 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-09-11T14:53:09.942 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:10.978 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (1036 ms) 2024-09-11T14:53:10.978 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-09-11T14:53:10.978 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:16.514 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (5537 ms) 2024-09-11T14:53:16.515 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-09-11T14:53:16.515 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:21.651 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (5136 ms) 2024-09-11T14:53:21.651 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-09-11T14:53:21.651 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:22.189 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (538 ms) 2024-09-11T14:53:22.189 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-09-11T14:53:22.189 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:22.667 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (478 ms) 2024-09-11T14:53:22.667 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-09-11T14:53:22.667 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:23.196 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (530 ms) 2024-09-11T14:53:23.197 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-09-11T14:53:23.197 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:23.708 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (512 ms) 2024-09-11T14:53:23.709 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-09-11T14:53:23.709 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:24.110 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:24.110 INFO:tasks.workunit.client.0.smithi059.stdout:aio discard: 0~1048576 = 0 2024-09-11T14:53:24.110 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:24.111 INFO:tasks.workunit.client.0.smithi059.stdout:aio discard: 0~4194304 = 0 2024-09-11T14:53:24.111 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:24.111 INFO:tasks.workunit.client.0.smithi059.stdout:aio discard: 3145728~1048576 = 0 2024-09-11T14:53:24.111 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (402 ms) 2024-09-11T14:53:24.111 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestFUA 2024-09-11T14:53:24.111 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:26.390 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.390 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.390 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.390 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.390 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.390 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:26.390 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.391 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.392 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:26.393 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:26.394 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:26.394 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.394 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:26.394 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:26.394 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:26.394 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.394 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:26.395 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:26.395 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:26.395 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.395 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:26.395 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:26.395 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:26.395 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestFUA (2279 ms) 2024-09-11T14:53:26.395 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-09-11T14:53:26.396 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:53:26.729 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.729 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.729 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.729 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.730 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:53:26.730 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.730 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.730 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.730 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.730 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.730 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.730 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.731 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.732 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.732 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:26.732 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:26.732 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:26.732 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:26.732 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:53:26.732 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:53:27.080 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:53:27.081 INFO:tasks.workunit.client.0.smithi059.stdout:start compare and write 2024-09-11T14:53:27.081 INFO:tasks.workunit.client.0.smithi059.stdout:compare and wrote: 512 2024-09-11T14:53:27.081 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.081 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:53:27.081 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:27.105 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:27.105 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:53:27.105 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.105 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:53:27.105 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:27.129 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:27.129 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:53:27.129 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.129 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:53:27.129 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:27.153 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:27.153 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:53:27.154 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.154 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:53:27.154 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:27.179 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:27.180 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:53:27.180 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.180 INFO:tasks.workunit.client.0.smithi059.stdout:started aio compare and write 2024-09-11T14:53:27.180 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:27.247 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:27.247 INFO:tasks.workunit.client.0.smithi059.stdout:finished aio compare and write 2024-09-11T14:53:27.247 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:27.248 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:27.248 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:27.248 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:27.248 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:27.248 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.248 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:27.248 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:27.253 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:27.253 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:53:27.254 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.254 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:27.254 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:27.260 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:27.260 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:53:27.260 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.260 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:27.260 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:27.266 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:27.266 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:53:27.266 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.266 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:27.266 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:27.272 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:27.272 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:53:27.272 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:27.272 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:53:27.273 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:53:27.273 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:53:27.273 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:53:27.273 INFO:tasks.workunit.client.0.smithi059.stdout:discard: 512~512 2024-09-11T14:53:27.273 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:51.522 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:51.522 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.523 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.524 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.525 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:53:51.526 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.526 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.526 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.526 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.526 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:53:51.526 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.526 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.527 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:53:51.527 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.527 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.527 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:53:51.527 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.527 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.527 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:53:51.527 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.528 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.529 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:53:51.529 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.529 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.529 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.529 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.529 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:51.529 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:51.529 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:51.530 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:51.745 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:51.745 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:51.745 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.745 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.746 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:51.746 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:51.746 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:51.894 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:51.894 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:51.895 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:52.567 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:52.567 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:52.567 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:52.567 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:52.567 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:52.567 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:52.567 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:53.007 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:53.007 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:53.008 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:53.008 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:53.008 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:53.008 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:53.008 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:53.868 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:53.868 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:53.868 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:53.868 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:53.868 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:53.868 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:53.868 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:54.466 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:54.466 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:54.466 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:54.466 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:54.466 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:54.466 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:54.466 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:54.467 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:54.467 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:54.467 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:54.467 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:54.467 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:54.467 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:55.895 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:55.896 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:55.896 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:55.896 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:55.896 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:55.896 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:55.896 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:56.754 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:56.755 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:56.755 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:56.755 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:56.755 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:56.755 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:56.755 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:58.472 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:58.472 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:58.472 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:58.472 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:58.472 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:58.472 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:58.472 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:53:59.436 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:53:59.436 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:53:59.436 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:53:59.437 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:01.385 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:01.385 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:54:01.385 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:54:01.385 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:54:01.385 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:01.385 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:01.385 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:02.608 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:02.608 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:54:02.609 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:54:02.609 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:54:02.609 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:02.609 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:02.609 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:04.933 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:04.933 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:54:04.933 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:54:04.933 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:54:04.933 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:04.933 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:04.933 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:06.495 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:06.495 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:54:06.495 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:54:06.495 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:54:06.495 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:06.495 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:06.495 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:54:06.496 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:06.496 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:06.496 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:54:06.496 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:06.496 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:06.496 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:09.089 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:09.089 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:54:09.089 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:54:09.089 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:54:09.090 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:09.090 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:09.090 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:10.694 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:10.694 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:54:10.694 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:54:10.694 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:54:10.694 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:10.694 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:10.694 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:13.773 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:13.774 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:54:13.774 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:54:13.774 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:54:13.774 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:13.774 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:13.774 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:15.705 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:15.705 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:54:15.705 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:54:15.705 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:54:15.705 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:15.705 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:15.705 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:54:15.705 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:15.706 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:54:15.706 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:54:15.706 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestIOPP (49315 ms) 2024-09-11T14:54:15.706 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-09-11T14:54:15.706 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:16.177 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (473 ms) 2024-09-11T14:54:16.177 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-09-11T14:54:16.177 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:16.629 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (452 ms) 2024-09-11T14:54:16.630 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-09-11T14:54:16.630 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:17.174 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (545 ms) 2024-09-11T14:54:17.175 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-09-11T14:54:17.175 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:17.612 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (437 ms) 2024-09-11T14:54:17.612 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-09-11T14:54:17.612 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:18.160 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (549 ms) 2024-09-11T14:54:18.161 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-09-11T14:54:18.161 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:18.570 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:18.650 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (490 ms) 2024-09-11T14:54:18.650 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-11T14:54:18.650 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:19.166 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (515 ms) 2024-09-11T14:54:19.166 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-11T14:54:19.166 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:19.657 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:19.745 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (580 ms) 2024-09-11T14:54:19.745 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-09-11T14:54:19.746 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:20.311 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (565 ms) 2024-09-11T14:54:20.311 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-09-11T14:54:20.311 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:20.719 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:20.801 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (490 ms) 2024-09-11T14:54:20.801 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-11T14:54:20.801 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:21.455 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (655 ms) 2024-09-11T14:54:21.456 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-11T14:54:21.456 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:21.922 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:22.041 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (585 ms) 2024-09-11T14:54:22.041 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-09-11T14:54:22.041 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:23.142 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (1101 ms) 2024-09-11T14:54:23.142 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-09-11T14:54:23.142 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:23.730 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:24.041 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (899 ms) 2024-09-11T14:54:24.041 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-09-11T14:54:24.041 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:25.032 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (990 ms) 2024-09-11T14:54:25.032 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-09-11T14:54:25.032 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:25.677 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:26.255 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (1224 ms) 2024-09-11T14:54:26.256 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-09-11T14:54:26.256 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:31.898 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (5642 ms) 2024-09-11T14:54:31.898 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-09-11T14:54:31.898 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:38.683 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:39.400 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (7502 ms) 2024-09-11T14:54:39.400 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-09-11T14:54:39.400 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:54:39.876 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:54:39.876 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:54:39.876 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:54:39.876 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:54:39.876 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:54:39.876 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:39.877 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:54:39.877 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:39.897 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:39.897 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:54:39.897 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:39.897 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:54:39.897 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:39.918 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:39.918 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:54:39.918 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:39.918 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:54:39.918 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:39.950 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:39.950 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:54:39.950 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:39.951 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:54:39.951 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:39.970 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:39.971 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:54:39.971 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:39.971 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:54:39.971 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:54:39.983 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:54:39.983 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:54:39.983 INFO:tasks.workunit.client.0.smithi059.stdout:read: 512 2024-09-11T14:54:39.983 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:39.983 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:54:39.983 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:54:39.991 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:54:39.991 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:54:39.991 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:39.991 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:54:39.991 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:54:39.998 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:54:39.998 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:54:39.999 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:39.999 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:54:39.999 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:54:40.005 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:54:40.005 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:54:40.005 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:40.005 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:54:40.005 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:54:40.011 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:54:40.011 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:54:40.011 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:54:40.011 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:54:40.012 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:55:05.485 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:55:05.485 INFO:tasks.workunit.client.0.smithi059.stdout:finished read 2024-09-11T14:55:05.485 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:55:05.485 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.485 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 0 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 16384 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.486 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 49152 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 65536 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.487 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 98304 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 114688 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:55:05.488 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 147456 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 163840 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:55:05.489 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 196608 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 212992 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.490 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:05.491 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:05.505 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:05.505 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:05.505 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.505 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.505 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:05.505 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:05.505 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:05.730 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:05.730 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:05.730 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.730 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.730 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:05.730 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:05.730 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:05.890 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:05.890 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:05.890 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:05.891 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:06.598 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:06.599 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:06.599 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:06.599 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:06.599 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:06.599 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:06.599 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:07.031 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:07.031 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:07.031 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:07.031 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:07.031 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:07.031 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:07.031 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:07.992 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:07.992 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:07.992 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:07.992 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:07.992 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:07.992 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:07.993 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:08.569 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:08.570 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:08.570 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:08.570 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:08.570 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:10.223 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:10.224 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:11.157 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:11.157 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:11.157 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:11.158 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:11.158 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:11.158 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:11.158 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:12.751 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:12.752 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:12.752 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:12.752 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:12.752 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:12.752 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:12.752 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:13.771 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:13.771 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:13.772 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:13.773 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:15.861 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:15.861 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:15.861 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:15.862 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:15.862 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:15.862 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:15.862 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:17.087 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:17.087 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:17.087 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:17.087 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:17.087 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:17.087 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:17.087 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:19.760 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:19.760 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:19.760 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:19.761 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:19.761 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:19.761 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:19.761 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:21.226 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:21.227 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:21.227 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:21.227 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:21.227 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:23.953 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:23.953 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:23.953 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:23.953 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:23.953 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:23.953 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:23.953 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:25.627 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:25.627 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:25.627 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:25.628 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:25.628 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:25.628 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:25.628 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:28.981 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:28.981 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:28.982 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:28.982 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:28.982 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:28.982 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:28.982 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:55:30.814 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:55:30.814 INFO:tasks.workunit.client.0.smithi059.stdout:finished writesame 2024-09-11T14:55:30.814 INFO:tasks.workunit.client.0.smithi059.stdout:to verify the data 2024-09-11T14:55:30.814 INFO:tasks.workunit.client.0.smithi059.stdout:verified 2024-09-11T14:55:30.814 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:30.815 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:30.815 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:30.815 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:55:30.815 INFO:tasks.workunit.client.0.smithi059.stdout:started writesame 2024-09-11T14:55:30.815 INFO:tasks.workunit.client.0.smithi059.stdout:expected fail, finished writesame 2024-09-11T14:55:30.815 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (51415 ms) 2024-09-11T14:55:30.815 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-09-11T14:55:30.815 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:55:33.184 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 0 2024-09-11T14:55:33.184 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T14:55:33.184 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 1 2024-09-11T14:55:33.184 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:55:33.184 INFO:tasks.workunit.client.0.smithi059.stdout:snap: orig 2024-09-11T14:55:33.184 INFO:tasks.workunit.client.0.smithi059.stdout:found orig with size 2097152 2024-09-11T14:55:33.184 INFO:tasks.workunit.client.0.smithi059.stdout:read: 80 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:write test data! 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 80 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 2 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:snap: orig 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:snap: written 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:found orig with size 2097152 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:found written with size 2097152 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:read: 80 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:read: 80 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:read: 80 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:write to snapshot returned -30 2024-09-11T14:55:33.185 INFO:tasks.workunit.client.0.smithi059.stdout:Read-only file system 2024-09-11T14:55:33.928 INFO:tasks.workunit.client.0.smithi059.stdout:read: 80 2024-09-11T14:55:33.928 INFO:tasks.workunit.client.0.smithi059.stdout:read: 80 2024-09-11T14:55:33.928 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 80 2024-09-11T14:55:33.928 INFO:tasks.workunit.client.0.smithi059.stdout:opening testimg@orig 2024-09-11T14:55:33.928 INFO:tasks.workunit.client.0.smithi059.stdout:read: 80 2024-09-11T14:55:33.928 INFO:tasks.workunit.client.0.smithi059.stdout:write to snapshot returned -30 2024-09-11T14:55:33.928 INFO:tasks.workunit.client.0.smithi059.stdout:Read-only file system 2024-09-11T14:55:36.138 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 2 2024-09-11T14:55:36.138 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:55:36.138 INFO:tasks.workunit.client.0.smithi059.stdout:snap: orig 2024-09-11T14:55:36.138 INFO:tasks.workunit.client.0.smithi059.stdout:snap: written 2024-09-11T14:55:36.138 INFO:tasks.workunit.client.0.smithi059.stdout:found orig with size 2097152 2024-09-11T14:55:36.138 INFO:tasks.workunit.client.0.smithi059.stdout:found written with size 2097152 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 1 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:snap: orig 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:found orig with size 2097152 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 0 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (5324 ms) 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-09-11T14:55:36.139 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:55:38.139 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (2000 ms) 2024-09-11T14:55:38.139 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestClone 2024-09-11T14:55:38.139 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:55:51.083 INFO:tasks.workunit.client.0.smithi059.stdout:made parent image "parent" 2024-09-11T14:55:51.083 INFO:tasks.workunit.client.0.smithi059.stdout:parent has no parent info 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:made snapshot "parent@parent_snap" 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:can't unprotect an unprotected snap 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:can't protect a protected snap 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:made and opened clone "child" 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:read: 8 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:read: 8 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:read: 8 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:sizes and overlaps are good between parent and child 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:child image successfully cloned all image-meta pairs 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:sized down clone, changed overlap 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:parent info: size 4194304 obj_size 4194304 parent_pool 18446744073709551615 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:sized up clone, changed size but not overlap or parent's size 2024-09-11T14:55:51.084 INFO:tasks.workunit.client.0.smithi059.stdout:can't remove parent while child still exists 2024-09-11T14:55:51.085 INFO:tasks.workunit.client.0.smithi059.stdout:can't remove parent while still protected 2024-09-11T14:55:51.085 INFO:tasks.workunit.client.0.smithi059.stdout:removed parent snap after unprotecting 2024-09-11T14:55:51.085 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestClone (12945 ms) 2024-09-11T14:55:51.085 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestClone2 2024-09-11T14:55:51.085 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:01.301 INFO:tasks.workunit.client.0.smithi059.stdout:made parent image "parent" 2024-09-11T14:56:01.301 INFO:tasks.workunit.client.0.smithi059.stdout:parent has no parent info 2024-09-11T14:56:01.301 INFO:tasks.workunit.client.0.smithi059.stdout:made snapshot "parent@parent_snap" 2024-09-11T14:56:01.301 INFO:tasks.workunit.client.0.smithi059.stdout:made and opened clone "child" 2024-09-11T14:56:01.301 INFO:tasks.workunit.client.0.smithi059.stdout:child image successfully cloned all image-meta pairs 2024-09-11T14:56:01.302 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestClone2 (10217 ms) 2024-09-11T14:56:01.302 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ListChildren 2024-09-11T14:56:04.610 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:23.731 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.731 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 1 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:child: image124 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124/10693854a0ae 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.732 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 2 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:child: image125 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:child: image124 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124/10693854a0ae 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.733 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125/1069ee27ae66 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.734 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 3 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 3 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:child: image125 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:child: image124 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:child: image126 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124/10693854a0ae 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125/1069ee27ae66 2024-09-11T14:56:23.735 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image126/106971301b3c 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.736 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 3 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 3 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:child: image125 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:child: image124 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:child: image126 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124/10693854a0ae 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125/1069ee27ae66 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image126/106971301b3c 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.737 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.738 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 4 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 4 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:child: image125 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:child: image124 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:child: image126 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:child: image127 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124/10693854a0ae 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125/1069ee27ae66 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image126/106971301b3c 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image127/1069b8a839bb 2024-09-11T14:56:23.739 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.740 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:23.741 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:23.741 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:23.741 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:23.741 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:23.741 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.741 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:23.741 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:23.741 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.194 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image124 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 4 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 4 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:child: image125 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:child: image124 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:child: image126 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:child: image127 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image124/10693854a0ae 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125/1069ee27ae66 2024-09-11T14:56:34.195 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image126/106971301b3c 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image127/1069b8a839bb 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:34.196 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image126 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 3 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 3 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:child: image125 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:child: image126 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:child: image127 2024-09-11T14:56:34.197 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125/1069ee27ae66 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image126/106971301b3c 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image127/1069b8a839bb 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image127 2024-09-11T14:56:34.198 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 2 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 2 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:child: image125 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:child: image127 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125/1069ee27ae66 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-17/image127/1069b8a839bb 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:trying to find test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:checking test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.199 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 1 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 1 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:child: image125 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-16/image125/1069ee27ae66 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 0 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ListChildren (32894 ms) 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:6995: Skipped 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:Not supported by crimson yet. Skipped 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered (0 ms) 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.LockingPP 2024-09-11T14:56:34.200 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:35.023 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.LockingPP (828 ms) 2024-09-11T14:56:35.023 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.FlushAio 2024-09-11T14:56:35.023 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:43.856 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.FlushAio (8833 ms) 2024-09-11T14:56:43.856 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-09-11T14:56:43.856 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:49.235 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.FlushAioPP (5379 ms) 2024-09-11T14:56:49.235 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-09-11T14:56:49.235 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:49.623 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (388 ms) 2024-09-11T14:56:49.623 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-09-11T14:56:49.623 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:50.129 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (507 ms) 2024-09-11T14:56:50.130 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-09-11T14:56:50.130 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:50.422 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ZeroLengthRead (292 ms) 2024-09-11T14:56:50.422 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-09-11T14:56:50.422 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:51.025 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.LargeCacheRead (603 ms) 2024-09-11T14:56:51.025 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-09-11T14:56:51.025 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:56:57.960 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestPendingAio (6936 ms) 2024-09-11T14:56:57.960 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-09-11T14:56:57.961 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:00.449 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:05.412 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyup (7452 ms) 2024-09-11T14:57:05.412 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-09-11T14:57:05.413 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:08.467 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:12.427 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyupMismatch (7014 ms) 2024-09-11T14:57:12.427 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.Flatten 2024-09-11T14:57:12.427 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:15.520 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.Flatten (3094 ms) 2024-09-11T14:57:15.520 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.Sparsify 2024-09-11T14:57:15.521 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:16.232 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 32768 2024-09-11T14:57:16.232 INFO:tasks.workunit.client.0.smithi059.stdout:read: 32768 2024-09-11T14:57:16.232 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.Sparsify (711 ms) 2024-09-11T14:57:16.232 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-09-11T14:57:16.232 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:16.840 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SparsifyPP (607 ms) 2024-09-11T14:57:16.840 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-09-11T14:57:16.840 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:22.124 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapshotLimit (5286 ms) 2024-09-11T14:57:22.125 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-09-11T14:57:22.125 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:28.104 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5979 ms) 2024-09-11T14:57:28.104 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-09-11T14:57:28.104 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:29.230 INFO:tasks.workunit.client.0.smithi059.stdout:0.00% done 2024-09-11T14:57:29.230 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.RebuildObjectMapViaLockOwner (1127 ms) 2024-09-11T14:57:29.230 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-09-11T14:57:29.230 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:30.490 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.RenameViaLockOwner (1259 ms) 2024-09-11T14:57:30.490 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-09-11T14:57:30.490 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:32.441 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapCreateViaLockOwner (1952 ms) 2024-09-11T14:57:32.441 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-09-11T14:57:32.442 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:36.346 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapRemoveViaLockOwner (3904 ms) 2024-09-11T14:57:36.346 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-09-11T14:57:36.346 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:37.398 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.UpdateFeaturesViaLockOwner (1053 ms) 2024-09-11T14:57:37.399 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-11T14:57:37.399 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:9736: Skipped 2024-09-11T14:57:37.399 INFO:tasks.workunit.client.0.smithi059.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:57:37.399 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-09-11T14:57:37.399 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-09-11T14:57:37.399 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:42.323 INFO:tasks.workunit.client.0.smithi059.stdout:100.00% done 2024-09-11T14:57:42.323 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapRemove2 (4925 ms) 2024-09-11T14:57:42.323 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-09-11T14:57:42.323 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:44.520 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapRenameViaLockOwner (2196 ms) 2024-09-11T14:57:44.520 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-09-11T14:57:44.520 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:46.537 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapProtectViaLockOwner (2017 ms) 2024-09-11T14:57:46.537 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-09-11T14:57:46.537 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:48.875 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapUnprotectViaLockOwner (2338 ms) 2024-09-11T14:57:48.875 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-09-11T14:57:48.875 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:52.227 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.FlattenViaLockOwner (3352 ms) 2024-09-11T14:57:52.227 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-09-11T14:57:52.227 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:53.066 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ResizeViaLockOwner (840 ms) 2024-09-11T14:57:53.066 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-09-11T14:57:53.067 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:57:53.754 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SparsifyViaLockOwner (688 ms) 2024-09-11T14:57:53.755 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-09-11T14:57:53.755 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:23.658 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ObjectMapConsistentSnap (29903 ms) 2024-09-11T14:58:23.658 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.Metadata 2024-09-11T14:58:23.658 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:27.269 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.Metadata (3611 ms) 2024-09-11T14:58:27.269 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-09-11T14:58:27.269 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:32.019 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.MetadataPP (4750 ms) 2024-09-11T14:58:32.019 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-09-11T14:58:32.019 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:34.764 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.UpdateFeatures (2745 ms) 2024-09-11T14:58:34.764 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-09-11T14:58:34.765 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2024-09-11T14:58:34.765 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-09-11T14:58:34.765 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:36.938 INFO:tasks.workunit.client.0.smithi059.stdout:0.00% done 2024-09-11T14:58:36.938 INFO:tasks.workunit.client.0.smithi059.stdout:25.00% done 2024-09-11T14:58:36.938 INFO:tasks.workunit.client.0.smithi059.stdout:50.00% done 2024-09-11T14:58:36.938 INFO:tasks.workunit.client.0.smithi059.stdout:75.00% done 2024-09-11T14:58:36.938 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.RebuildObjectMap (2173 ms) 2024-09-11T14:58:36.938 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-09-11T14:58:38.052 INFO:tasks.workunit.client.0.smithi059.stdout:0.00% done 2024-09-11T14:58:38.052 INFO:tasks.workunit.client.0.smithi059.stdout:25.00% done 2024-09-11T14:58:38.052 INFO:tasks.workunit.client.0.smithi059.stdout:50.00% done 2024-09-11T14:58:38.052 INFO:tasks.workunit.client.0.smithi059.stdout:75.00% done 2024-09-11T14:58:38.052 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.RebuildNewObjectMap (1114 ms) 2024-09-11T14:58:38.053 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-09-11T14:58:38.053 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:40.934 INFO:tasks.workunit.client.0.smithi059.stdout:0.00% done 2024-09-11T14:58:40.935 INFO:tasks.workunit.client.0.smithi059.stdout:25.00% done 2024-09-11T14:58:40.935 INFO:tasks.workunit.client.0.smithi059.stdout:50.00% done 2024-09-11T14:58:40.935 INFO:tasks.workunit.client.0.smithi059.stdout:75.00% done 2024-09-11T14:58:40.935 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.CheckObjectMap (2883 ms) 2024-09-11T14:58:40.935 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-09-11T14:58:40.935 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:41.481 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.BlockingAIO (547 ms) 2024-09-11T14:58:41.482 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-09-11T14:58:41.482 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:46.552 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ExclusiveLockTransition (5070 ms) 2024-09-11T14:58:46.552 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-09-11T14:58:46.553 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:10793: Skipped 2024-09-11T14:58:46.553 INFO:tasks.workunit.client.0.smithi059.stdout:Skipping due to unmet REQUIRE 2024-09-11T14:58:46.553 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-09-11T14:58:46.553 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-09-11T14:58:46.553 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:48.981 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.CacheMayCopyOnWrite (2430 ms) 2024-09-11T14:58:48.981 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-09-11T14:58:48.981 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:51.192 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (2210 ms) 2024-09-11T14:58:51.192 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-09-11T14:58:55.576 INFO:tasks.workunit.client.0.smithi059.stdout:0.00% done 2024-09-11T14:58:55.577 INFO:tasks.workunit.client.0.smithi059.stdout:100.00% done 2024-09-11T14:58:55.577 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestImageOptions (4385 ms) 2024-09-11T14:58:55.577 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-09-11T14:58:58.629 INFO:tasks.workunit.client.0.smithi059.stdout:0.00% done 2024-09-11T14:58:58.629 INFO:tasks.workunit.client.0.smithi059.stdout:100.00% done 2024-09-11T14:58:58.629 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (3052 ms) 2024-09-11T14:58:58.629 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-09-11T14:58:58.630 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2024-09-11T14:58:58.630 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-09-11T14:58:58.630 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-09-11T14:58:58.630 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-09-11T14:58:58.630 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:59.404 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:58:59.404 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:58:59.404 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:58:59.404 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:58:59.404 INFO:tasks.workunit.client.0.smithi059.stdout:wrote: 512 2024-09-11T14:58:59.404 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:58:59.405 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:started write 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:write completion cb called! 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 0 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:finished write 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.406 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:58:59.407 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:58:59.408 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:58:59.408 INFO:tasks.workunit.client.0.smithi059.stdout:created completion 2024-09-11T14:58:59.408 INFO:tasks.workunit.client.0.smithi059.stdout:started read 2024-09-11T14:58:59.408 INFO:tasks.workunit.client.0.smithi059.stdout:read completion cb called! 2024-09-11T14:58:59.408 INFO:tasks.workunit.client.0.smithi059.stdout:return value is: 512 2024-09-11T14:58:59.408 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ImagePollIO (776 ms) 2024-09-11T14:58:59.408 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.Mirror 2024-09-11T14:58:59.517 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:58:59.653 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:04.367 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.Mirror (4963 ms) 2024-09-11T14:59:04.368 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-09-11T14:59:04.558 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (191 ms) 2024-09-11T14:59:04.559 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-09-11T14:59:14.341 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.CreateWithMirrorEnabled (9782 ms) 2024-09-11T14:59:14.341 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-09-11T14:59:14.341 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:19.313 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (4973 ms) 2024-09-11T14:59:19.314 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-09-11T14:59:19.314 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:20.263 INFO:tasks.workunit.client.0.smithi059.stdout:1: acquiring exclusive lock 2024-09-11T14:59:20.263 INFO:tasks.workunit.client.0.smithi059.stdout:2: acquiring exclusive lock 2024-09-11T14:59:20.377 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock acquired 2024-09-11T14:59:20.420 INFO:tasks.workunit.client.0.smithi059.stdout:2: releasing exclusive lock 2024-09-11T14:59:20.501 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock released 2024-09-11T14:59:20.501 INFO:tasks.workunit.client.0.smithi059.stdout:2: acquiring exclusive lock 2024-09-11T14:59:20.606 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock acquired 2024-09-11T14:59:20.652 INFO:tasks.workunit.client.0.smithi059.stdout:1: releasing exclusive lock 2024-09-11T14:59:20.720 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock released 2024-09-11T14:59:20.721 INFO:tasks.workunit.client.0.smithi059.stdout:1: acquiring exclusive lock 2024-09-11T14:59:20.832 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock acquired 2024-09-11T14:59:20.840 INFO:tasks.workunit.client.0.smithi059.stdout:2: releasing exclusive lock 2024-09-11T14:59:20.918 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock released 2024-09-11T14:59:20.918 INFO:tasks.workunit.client.0.smithi059.stdout:2: acquiring exclusive lock 2024-09-11T14:59:21.022 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock acquired 2024-09-11T14:59:21.028 INFO:tasks.workunit.client.0.smithi059.stdout:1: releasing exclusive lock 2024-09-11T14:59:21.129 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock released 2024-09-11T14:59:21.129 INFO:tasks.workunit.client.0.smithi059.stdout:1: acquiring exclusive lock 2024-09-11T14:59:21.303 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock acquired 2024-09-11T14:59:21.342 INFO:tasks.workunit.client.0.smithi059.stdout:2: releasing exclusive lock 2024-09-11T14:59:21.432 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock released 2024-09-11T14:59:21.432 INFO:tasks.workunit.client.0.smithi059.stdout:2: acquiring exclusive lock 2024-09-11T14:59:21.544 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock acquired 2024-09-11T14:59:21.584 INFO:tasks.workunit.client.0.smithi059.stdout:2: releasing exclusive lock 2024-09-11T14:59:21.661 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock released 2024-09-11T14:59:21.661 INFO:tasks.workunit.client.0.smithi059.stdout:2: acquiring exclusive lock 2024-09-11T14:59:21.745 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock acquired 2024-09-11T14:59:21.783 INFO:tasks.workunit.client.0.smithi059.stdout:1: releasing exclusive lock 2024-09-11T14:59:21.856 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock released 2024-09-11T14:59:21.856 INFO:tasks.workunit.client.0.smithi059.stdout:1: acquiring exclusive lock 2024-09-11T14:59:21.976 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock acquired 2024-09-11T14:59:22.000 INFO:tasks.workunit.client.0.smithi059.stdout:1: releasing exclusive lock 2024-09-11T14:59:22.089 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock released 2024-09-11T14:59:22.089 INFO:tasks.workunit.client.0.smithi059.stdout:1: acquiring exclusive lock 2024-09-11T14:59:22.199 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock acquired 2024-09-11T14:59:22.201 INFO:tasks.workunit.client.0.smithi059.stdout:2: releasing exclusive lock 2024-09-11T14:59:22.281 INFO:tasks.workunit.client.0.smithi059.stdout:2: exclusive lock released 2024-09-11T14:59:22.370 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock acquired 2024-09-11T14:59:22.396 INFO:tasks.workunit.client.0.smithi059.stdout:1: releasing exclusive lock 2024-09-11T14:59:22.452 INFO:tasks.workunit.client.0.smithi059.stdout:1: exclusive lock released 2024-09-11T14:59:22.749 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ExclusiveLock (3435 ms) 2024-09-11T14:59:22.749 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.BreakLock 2024-09-11T14:59:22.749 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:11553: Skipped 2024-09-11T14:59:22.749 INFO:tasks.workunit.client.0.smithi059.stdout:Not supported by crimson yet. Skipped 2024-09-11T14:59:22.749 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.BreakLock (0 ms) 2024-09-11T14:59:22.749 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-09-11T14:59:22.749 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:23.165 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (417 ms) 2024-09-11T14:59:23.165 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-09-11T14:59:23.166 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-09-11T14:59:23.166 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-09-11T14:59:23.166 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:24.490 INFO:tasks.workunit.client.0.smithi059.stdout:0.00% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:1.56% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:3.12% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:4.69% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:6.25% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:7.81% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:9.38% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:10.94% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:12.50% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:14.06% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:15.62% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:17.19% done 2024-09-11T14:59:24.491 INFO:tasks.workunit.client.0.smithi059.stdout:18.75% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:20.31% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:21.88% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:23.44% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:25.00% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:26.56% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:28.12% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:29.69% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:31.25% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:32.81% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:34.38% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:35.94% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:37.50% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:39.06% done 2024-09-11T14:59:24.492 INFO:tasks.workunit.client.0.smithi059.stdout:40.62% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:42.19% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:43.75% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:45.31% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:46.88% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:48.44% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:50.00% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:51.56% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:53.12% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:54.69% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:56.25% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:57.81% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:59.38% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:60.94% done 2024-09-11T14:59:24.493 INFO:tasks.workunit.client.0.smithi059.stdout:62.50% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:64.06% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:65.62% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:67.19% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:68.75% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:70.31% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:71.88% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:73.44% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:75.00% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:76.56% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:78.12% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:79.69% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:81.25% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:82.81% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:84.38% done 2024-09-11T14:59:24.494 INFO:tasks.workunit.client.0.smithi059.stdout:85.94% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:87.50% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:89.06% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:90.62% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:92.19% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:93.75% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:95.31% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:96.88% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:98.44% done 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurge (1325 ms) 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-09-11T14:59:24.495 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:25.681 INFO:tasks.workunit.client.0.smithi059.stdout:0.00% done 2024-09-11T14:59:25.681 INFO:tasks.workunit.client.0.smithi059.stdout:1.56% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:3.12% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:4.69% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:6.25% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:7.81% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:9.38% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:10.94% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:12.50% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:14.06% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:15.62% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:17.19% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:18.75% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:20.31% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:21.88% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:23.44% done 2024-09-11T14:59:25.682 INFO:tasks.workunit.client.0.smithi059.stdout:25.00% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:26.56% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:28.12% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:29.69% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:31.25% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:32.81% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:34.38% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:35.94% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:37.50% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:39.06% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:40.62% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:42.19% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:43.75% done 2024-09-11T14:59:25.683 INFO:tasks.workunit.client.0.smithi059.stdout:45.31% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:46.88% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:48.44% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:50.00% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:51.56% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:53.12% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:54.69% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:56.25% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:57.81% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:59.38% done 2024-09-11T14:59:25.684 INFO:tasks.workunit.client.0.smithi059.stdout:60.94% done 2024-09-11T14:59:25.685 INFO:tasks.workunit.client.0.smithi059.stdout:62.50% done 2024-09-11T14:59:25.685 INFO:tasks.workunit.client.0.smithi059.stdout:64.06% done 2024-09-11T14:59:25.685 INFO:tasks.workunit.client.0.smithi059.stdout:65.62% done 2024-09-11T14:59:25.685 INFO:tasks.workunit.client.0.smithi059.stdout:67.19% done 2024-09-11T14:59:25.685 INFO:tasks.workunit.client.0.smithi059.stdout:68.75% done 2024-09-11T14:59:25.685 INFO:tasks.workunit.client.0.smithi059.stdout:70.31% done 2024-09-11T14:59:25.686 INFO:tasks.workunit.client.0.smithi059.stdout:71.88% done 2024-09-11T14:59:25.686 INFO:tasks.workunit.client.0.smithi059.stdout:73.44% done 2024-09-11T14:59:25.686 INFO:tasks.workunit.client.0.smithi059.stdout:75.00% done 2024-09-11T14:59:25.686 INFO:tasks.workunit.client.0.smithi059.stdout:76.56% done 2024-09-11T14:59:25.686 INFO:tasks.workunit.client.0.smithi059.stdout:78.12% done 2024-09-11T14:59:25.687 INFO:tasks.workunit.client.0.smithi059.stdout:79.69% done 2024-09-11T14:59:25.687 INFO:tasks.workunit.client.0.smithi059.stdout:81.25% done 2024-09-11T14:59:25.687 INFO:tasks.workunit.client.0.smithi059.stdout:82.81% done 2024-09-11T14:59:25.687 INFO:tasks.workunit.client.0.smithi059.stdout:84.38% done 2024-09-11T14:59:25.687 INFO:tasks.workunit.client.0.smithi059.stdout:85.94% done 2024-09-11T14:59:25.687 INFO:tasks.workunit.client.0.smithi059.stdout:87.50% done 2024-09-11T14:59:25.688 INFO:tasks.workunit.client.0.smithi059.stdout:89.06% done 2024-09-11T14:59:25.688 INFO:tasks.workunit.client.0.smithi059.stdout:90.62% done 2024-09-11T14:59:25.688 INFO:tasks.workunit.client.0.smithi059.stdout:92.19% done 2024-09-11T14:59:25.688 INFO:tasks.workunit.client.0.smithi059.stdout:93.75% done 2024-09-11T14:59:25.688 INFO:tasks.workunit.client.0.smithi059.stdout:95.31% done 2024-09-11T14:59:25.689 INFO:tasks.workunit.client.0.smithi059.stdout:96.88% done 2024-09-11T14:59:25.689 INFO:tasks.workunit.client.0.smithi059.stdout:98.44% done 2024-09-11T14:59:25.689 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (1191 ms) 2024-09-11T14:59:25.689 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-09-11T14:59:25.689 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:25.824 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:28.542 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestTrashPurge (2860 ms) 2024-09-11T14:59:28.542 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-09-11T14:59:28.542 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:29.520 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestTrashMoveAndRestore (978 ms) 2024-09-11T14:59:29.520 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-09-11T14:59:29.520 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:30.118 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestListWatchers (598 ms) 2024-09-11T14:59:30.119 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-09-11T14:59:30.119 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:31.319 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestSetSnapById (1201 ms) 2024-09-11T14:59:31.319 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.Namespaces 2024-09-11T14:59:31.838 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.Namespaces (519 ms) 2024-09-11T14:59:31.838 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-09-11T14:59:32.478 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:32.523 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:34.105 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.NamespacesPP (2267 ms) 2024-09-11T14:59:34.105 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.Migration 2024-09-11T14:59:34.105 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:34.105 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:43.037 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.Migration (8933 ms) 2024-09-11T14:59:43.038 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-09-11T14:59:43.038 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:43.038 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:51.693 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.MigrationPP (8655 ms) 2024-09-11T14:59:51.693 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-09-11T14:59:51.693 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:51.973 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestGetAccessTimestamp (280 ms) 2024-09-11T14:59:51.973 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-09-11T14:59:51.973 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:52.289 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.TestGetModifyTimestamp (317 ms) 2024-09-11T14:59:52.290 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-09-11T14:59:52.290 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:55.562 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ZeroOverlapFlatten (3272 ms) 2024-09-11T14:59:55.562 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-09-11T14:59:55.968 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.PoolMetadata (406 ms) 2024-09-11T14:59:55.968 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-09-11T14:59:56.359 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.PoolMetadataPP (391 ms) 2024-09-11T14:59:56.359 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.Config 2024-09-11T14:59:56.401 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:57.084 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.Config (725 ms) 2024-09-11T14:59:57.084 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-09-11T14:59:57.121 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T14:59:57.841 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ConfigPP (758 ms) 2024-09-11T14:59:57.841 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-09-11T14:59:59.567 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:00:03.143 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:00:03.300 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:00:03.502 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:00:05.949 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.PoolStatsPP (8107 ms) 2024-09-11T15:00:05.949 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-09-11T15:00:07.599 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:00:18.034 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.ImageSpec (12085 ms) 2024-09-11T15:00:18.034 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-09-11T15:00:19.627 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:00:43.738 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 3 2024-09-11T15:00:43.738 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 3 2024-09-11T15:00:43.738 INFO:tasks.workunit.client.0.smithi059.stdout:child: image223 2024-09-11T15:00:43.738 INFO:tasks.workunit.client.0.smithi059.stdout:child: image225 2024-09-11T15:00:43.738 INFO:tasks.workunit.client.0.smithi059.stdout:child: image224 2024-09-11T15:00:43.738 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-1/image223/1069269eac45 2024-09-11T15:00:43.738 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T15:00:43.738 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-20/image224/10697cf26440 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout:found child test-librbd-smithi059-29996-1/image225/1069cd0c824c 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout:num children is: 0 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout:num snaps is: 0 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout:expected: 0 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.SnapRemoveWithChildMissing (25704 ms) 2024-09-11T15:00:43.739 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-09-11T15:00:45.541 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:01:00.363 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.QuiesceWatch (16624 ms) 2024-09-11T15:01:00.364 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-09-11T15:01:00.364 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:01:09.534 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (9173 ms) 2024-09-11T15:01:09.534 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-09-11T15:01:09.535 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:13115: Skipped 2024-09-11T15:01:09.535 INFO:tasks.workunit.client.0.smithi059.stdout:Not supported by crimson yet. Skipped 2024-09-11T15:01:09.535 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError (0 ms) 2024-09-11T15:01:09.535 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-09-11T15:01:09.535 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:01:09.779 INFO:tasks.workunit.client.0.smithi059.stderr:test quiesce is not long enough to time out 2024-09-11T15:01:19.367 INFO:tasks.workunit.client.0.smithi059.stderr:test quiesce is timed out 2024-09-11T15:01:19.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 0 2024-09-11T15:01:20.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 1 2024-09-11T15:01:21.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 2 2024-09-11T15:01:22.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 3 2024-09-11T15:01:23.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 4 2024-09-11T15:01:24.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 5 2024-09-11T15:01:25.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 6 2024-09-11T15:01:26.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 7 2024-09-11T15:01:27.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 8 2024-09-11T15:01:28.467 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 9 2024-09-11T15:01:29.468 INFO:tasks.workunit.client.0.smithi059.stderr:waiting for timed out ... 10 2024-09-11T15:01:40.468 INFO:tasks.workunit.client.0.smithi059.stderr:test retry succeeds 2024-09-11T15:01:44.751 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (35216 ms) 2024-09-11T15:01:44.751 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-09-11T15:01:44.751 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:01:45.488 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.WriteZeroes (737 ms) 2024-09-11T15:01:45.488 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-09-11T15:01:45.488 INFO:tasks.workunit.client.0.smithi059.stdout:using new format! 2024-09-11T15:01:46.140 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (653 ms) 2024-09-11T15:01:46.141 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.ConcurrentOperations 2024-09-11T15:01:46.141 INFO:tasks.workunit.client.0.smithi059.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4872-g0d61375b/rpm/el9/BUILD/ceph-19.3.0-4872-g0d61375b/src/test/librbd/test_librbd.cc:13474: Skipped 2024-09-11T15:01:46.141 INFO:tasks.workunit.client.0.smithi059.stdout:Not supported by crimson yet. Skipped 2024-09-11T15:01:46.141 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations (0 ms) 2024-09-11T15:01:46.141 INFO:tasks.workunit.client.0.smithi059.stdout:[ RUN ] TestLibRBD.FormatAndCloneFormatOptions 2024-09-11T15:01:58.044 INFO:tasks.workunit.client.0.smithi059.stdout:[ OK ] TestLibRBD.FormatAndCloneFormatOptions (11904 ms) 2024-09-11T15:02:17.532 INFO:tasks.workunit.client.0.smithi059.stdout:[----------] 171 tests from TestLibRBD (1316890 ms total) 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout:[----------] Global test environment tear-down 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout:[==========] 171 tests from 1 test suite ran. (1336428 ms total) 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout:[ PASSED ] 163 tests. 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] 7 tests, listed below: 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-11T15:02:17.533 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2024-09-11T15:02:17.534 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.BreakLock 2024-09-11T15:02:17.534 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError 2024-09-11T15:02:17.534 INFO:tasks.workunit.client.0.smithi059.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations 2024-09-11T15:02:17.534 INFO:tasks.workunit.client.0.smithi059.stdout:[ FAILED ] 1 test, listed below: 2024-09-11T15:02:17.534 INFO:tasks.workunit.client.0.smithi059.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 2024-09-11T15:02:17.534 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T15:02:17.534 INFO:tasks.workunit.client.0.smithi059.stdout: 1 FAILED TEST 2024-09-11T15:02:17.534 INFO:tasks.workunit.client.0.smithi059.stdout: YOU HAVE 1 DISABLED TEST 2024-09-11T15:02:17.535 INFO:tasks.workunit.client.0.smithi059.stdout: 2024-09-11T15:02:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T15:02:17.567 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2024-09-11T15:02:17.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T15:02:17.955 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-09-11T15:02:18.200 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6f077332b0e34d2e9264a30f779a20d2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-09-11T15:02:18.203 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T15:02:18.211 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T15:02:18.211 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T15:02:18.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:18.337+0000 7f3425191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:18.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:18.347+0000 7f3425191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:18.548 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T15:02:18.548 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-11T15:02:18.563 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":982,"stamp":"2024-09-11T15:02:17.518910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":68768,"kb_used_data":68768,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156824416,"statfs":{"total":160658620416,"available":160588201984,"internally_reserved":0,"allocated":70418432,"data_stored":70418432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5632,"num_objects":-72,"num_object_clones":0,"num_object_copies":-144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3894,"num_read_kb":-505,"num_write":-403,"num_write_kb":-259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.989710"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.639763+0000","last_change":"2024-09-11T14:39:10.591513+0000","last_active":"2024-09-11T15:02:15.639763+0000","last_peered":"2024-09-11T15:02:15.639763+0000","last_clean":"2024-09-11T15:02:15.639763+0000","last_became_active":"2024-09-11T14:39:08.649048+0000","last_became_peered":"2024-09-11T14:39:08.649048+0000","last_unstale":"2024-09-11T15:02:15.639763+0000","last_undegraded":"2024-09-11T15:02:15.639763+0000","last_fullsized":"2024-09-11T15:02:15.639763+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.581187+0000","last_change":"2024-09-11T14:39:10.590719+0000","last_active":"2024-09-11T15:02:15.581187+0000","last_peered":"2024-09-11T15:02:15.581187+0000","last_clean":"2024-09-11T15:02:15.581187+0000","last_became_active":"2024-09-11T14:39:08.671767+0000","last_became_peered":"2024-09-11T14:39:08.671767+0000","last_unstale":"2024-09-11T15:02:15.581187+0000","last_undegraded":"2024-09-11T15:02:15.581187+0000","last_fullsized":"2024-09-11T15:02:15.581187+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.612936+0000","last_change":"2024-09-11T14:39:10.595741+0000","last_active":"2024-09-11T15:02:15.612936+0000","last_peered":"2024-09-11T15:02:15.612936+0000","last_clean":"2024-09-11T15:02:15.612936+0000","last_became_active":"2024-09-11T14:39:08.728005+0000","last_became_peered":"2024-09-11T14:39:08.728005+0000","last_unstale":"2024-09-11T15:02:15.612936+0000","last_undegraded":"2024-09-11T15:02:15.612936+0000","last_fullsized":"2024-09-11T15:02:15.612936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.644307+0000","last_change":"2024-09-11T14:39:10.592063+0000","last_active":"2024-09-11T15:02:15.644307+0000","last_peered":"2024-09-11T15:02:15.644307+0000","last_clean":"2024-09-11T15:02:15.644307+0000","last_became_active":"2024-09-11T14:39:08.660946+0000","last_became_peered":"2024-09-11T14:39:08.660946+0000","last_unstale":"2024-09-11T15:02:15.644307+0000","last_undegraded":"2024-09-11T15:02:15.644307+0000","last_fullsized":"2024-09-11T15:02:15.644307+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":836,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-11T15:02:12.774111+0000","last_change":"2024-09-11T14:39:10.598829+0000","last_active":"2024-09-11T15:02:12.774111+0000","last_peered":"2024-09-11T15:02:12.774111+0000","last_clean":"2024-09-11T15:02:12.774111+0000","last_became_active":"2024-09-11T14:39:08.696312+0000","last_became_peered":"2024-09-11T14:39:08.696312+0000","last_unstale":"2024-09-11T15:02:12.774111+0000","last_undegraded":"2024-09-11T15:02:12.774111+0000","last_fullsized":"2024-09-11T15:02:12.774111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":843,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.665809+0000","last_change":"2024-09-11T14:39:10.595170+0000","last_active":"2024-09-11T15:02:15.665809+0000","last_peered":"2024-09-11T15:02:15.665809+0000","last_clean":"2024-09-11T15:02:15.665809+0000","last_became_active":"2024-09-11T14:39:08.722304+0000","last_became_peered":"2024-09-11T14:39:08.722304+0000","last_unstale":"2024-09-11T15:02:15.665809+0000","last_undegraded":"2024-09-11T15:02:15.665809+0000","last_fullsized":"2024-09-11T15:02:15.665809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":843,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.636417+0000","last_change":"2024-09-11T14:39:10.588768+0000","last_active":"2024-09-11T15:02:15.636417+0000","last_peered":"2024-09-11T15:02:15.636417+0000","last_clean":"2024-09-11T15:02:15.636417+0000","last_became_active":"2024-09-11T14:39:08.670002+0000","last_became_peered":"2024-09-11T14:39:08.670002+0000","last_unstale":"2024-09-11T15:02:15.636417+0000","last_undegraded":"2024-09-11T15:02:15.636417+0000","last_fullsized":"2024-09-11T15:02:15.636417+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":842,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.633111+0000","last_change":"2024-09-11T14:39:10.596036+0000","last_active":"2024-09-11T15:02:15.633111+0000","last_peered":"2024-09-11T15:02:15.633111+0000","last_clean":"2024-09-11T15:02:15.633111+0000","last_became_active":"2024-09-11T14:39:08.669437+0000","last_became_peered":"2024-09-11T14:39:08.669437+0000","last_unstale":"2024-09-11T15:02:15.633111+0000","last_undegraded":"2024-09-11T15:02:15.633111+0000","last_fullsized":"2024-09-11T15:02:15.633111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":910,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.636984+0000","last_change":"2024-09-11T14:39:07.488096+0000","last_active":"2024-09-11T15:02:15.636984+0000","last_peered":"2024-09-11T15:02:15.636984+0000","last_clean":"2024-09-11T15:02:15.636984+0000","last_became_active":"2024-09-11T14:39:07.454105+0000","last_became_peered":"2024-09-11T14:39:07.454105+0000","last_unstale":"2024-09-11T15:02:15.636984+0000","last_undegraded":"2024-09-11T15:02:15.636984+0000","last_fullsized":"2024-09-11T15:02:15.636984+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_clean_scrub_stamp":"2024-09-11T14:39:06.320059+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160591065088,"internally_reserved":0,"allocated":67555328,"data_stored":67555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654707043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":26560,"kb_used_data":26560,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52271168,"statfs":{"total":53552873472,"available":53525676032,"internally_reserved":0,"allocated":27197440,"data_stored":27197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739750,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":10940,"kb_used_data":10940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52286788,"statfs":{"total":53552873472,"available":53541670912,"internally_reserved":0,"allocated":11202560,"data_stored":11202560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":31268,"kb_used_data":31268,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52266460,"statfs":{"total":53552873472,"available":53520855040,"internally_reserved":0,"allocated":32018432,"data_stored":32018432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53521772544,"internally_reserved":0,"allocated":31100928,"data_stored":31100928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53526876160,"internally_reserved":0,"allocated":25997312,"data_stored":25997312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T15:02:18.564 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T15:02:18.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:18.680+0000 7feb488fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:18.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:18.688+0000 7feb488fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:18.893 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T15:02:18.894 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-11T15:02:18.909 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":982,"stamp":"2024-09-11T15:02:17.518910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":68768,"kb_used_data":68768,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156824416,"statfs":{"total":160658620416,"available":160588201984,"internally_reserved":0,"allocated":70418432,"data_stored":70418432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5632,"num_objects":-72,"num_object_clones":0,"num_object_copies":-144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3894,"num_read_kb":-505,"num_write":-403,"num_write_kb":-259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.989710"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.639763+0000","last_change":"2024-09-11T14:39:10.591513+0000","last_active":"2024-09-11T15:02:15.639763+0000","last_peered":"2024-09-11T15:02:15.639763+0000","last_clean":"2024-09-11T15:02:15.639763+0000","last_became_active":"2024-09-11T14:39:08.649048+0000","last_became_peered":"2024-09-11T14:39:08.649048+0000","last_unstale":"2024-09-11T15:02:15.639763+0000","last_undegraded":"2024-09-11T15:02:15.639763+0000","last_fullsized":"2024-09-11T15:02:15.639763+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.581187+0000","last_change":"2024-09-11T14:39:10.590719+0000","last_active":"2024-09-11T15:02:15.581187+0000","last_peered":"2024-09-11T15:02:15.581187+0000","last_clean":"2024-09-11T15:02:15.581187+0000","last_became_active":"2024-09-11T14:39:08.671767+0000","last_became_peered":"2024-09-11T14:39:08.671767+0000","last_unstale":"2024-09-11T15:02:15.581187+0000","last_undegraded":"2024-09-11T15:02:15.581187+0000","last_fullsized":"2024-09-11T15:02:15.581187+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.612936+0000","last_change":"2024-09-11T14:39:10.595741+0000","last_active":"2024-09-11T15:02:15.612936+0000","last_peered":"2024-09-11T15:02:15.612936+0000","last_clean":"2024-09-11T15:02:15.612936+0000","last_became_active":"2024-09-11T14:39:08.728005+0000","last_became_peered":"2024-09-11T14:39:08.728005+0000","last_unstale":"2024-09-11T15:02:15.612936+0000","last_undegraded":"2024-09-11T15:02:15.612936+0000","last_fullsized":"2024-09-11T15:02:15.612936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.644307+0000","last_change":"2024-09-11T14:39:10.592063+0000","last_active":"2024-09-11T15:02:15.644307+0000","last_peered":"2024-09-11T15:02:15.644307+0000","last_clean":"2024-09-11T15:02:15.644307+0000","last_became_active":"2024-09-11T14:39:08.660946+0000","last_became_peered":"2024-09-11T14:39:08.660946+0000","last_unstale":"2024-09-11T15:02:15.644307+0000","last_undegraded":"2024-09-11T15:02:15.644307+0000","last_fullsized":"2024-09-11T15:02:15.644307+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":836,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-09-11T15:02:12.774111+0000","last_change":"2024-09-11T14:39:10.598829+0000","last_active":"2024-09-11T15:02:12.774111+0000","last_peered":"2024-09-11T15:02:12.774111+0000","last_clean":"2024-09-11T15:02:12.774111+0000","last_became_active":"2024-09-11T14:39:08.696312+0000","last_became_peered":"2024-09-11T14:39:08.696312+0000","last_unstale":"2024-09-11T15:02:12.774111+0000","last_undegraded":"2024-09-11T15:02:12.774111+0000","last_fullsized":"2024-09-11T15:02:12.774111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":843,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.665809+0000","last_change":"2024-09-11T14:39:10.595170+0000","last_active":"2024-09-11T15:02:15.665809+0000","last_peered":"2024-09-11T15:02:15.665809+0000","last_clean":"2024-09-11T15:02:15.665809+0000","last_became_active":"2024-09-11T14:39:08.722304+0000","last_became_peered":"2024-09-11T14:39:08.722304+0000","last_unstale":"2024-09-11T15:02:15.665809+0000","last_undegraded":"2024-09-11T15:02:15.665809+0000","last_fullsized":"2024-09-11T15:02:15.665809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":843,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.636417+0000","last_change":"2024-09-11T14:39:10.588768+0000","last_active":"2024-09-11T15:02:15.636417+0000","last_peered":"2024-09-11T15:02:15.636417+0000","last_clean":"2024-09-11T15:02:15.636417+0000","last_became_active":"2024-09-11T14:39:08.670002+0000","last_became_peered":"2024-09-11T14:39:08.670002+0000","last_unstale":"2024-09-11T15:02:15.636417+0000","last_undegraded":"2024-09-11T15:02:15.636417+0000","last_fullsized":"2024-09-11T15:02:15.636417+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":842,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.633111+0000","last_change":"2024-09-11T14:39:10.596036+0000","last_active":"2024-09-11T15:02:15.633111+0000","last_peered":"2024-09-11T15:02:15.633111+0000","last_clean":"2024-09-11T15:02:15.633111+0000","last_became_active":"2024-09-11T14:39:08.669437+0000","last_became_peered":"2024-09-11T14:39:08.669437+0000","last_unstale":"2024-09-11T15:02:15.633111+0000","last_undegraded":"2024-09-11T15:02:15.633111+0000","last_fullsized":"2024-09-11T15:02:15.633111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":910,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.636984+0000","last_change":"2024-09-11T14:39:07.488096+0000","last_active":"2024-09-11T15:02:15.636984+0000","last_peered":"2024-09-11T15:02:15.636984+0000","last_clean":"2024-09-11T15:02:15.636984+0000","last_became_active":"2024-09-11T14:39:07.454105+0000","last_became_peered":"2024-09-11T14:39:07.454105+0000","last_unstale":"2024-09-11T15:02:15.636984+0000","last_undegraded":"2024-09-11T15:02:15.636984+0000","last_fullsized":"2024-09-11T15:02:15.636984+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_clean_scrub_stamp":"2024-09-11T14:39:06.320059+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160591065088,"internally_reserved":0,"allocated":67555328,"data_stored":67555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654707043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":26560,"kb_used_data":26560,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52271168,"statfs":{"total":53552873472,"available":53525676032,"internally_reserved":0,"allocated":27197440,"data_stored":27197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739750,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":10940,"kb_used_data":10940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52286788,"statfs":{"total":53552873472,"available":53541670912,"internally_reserved":0,"allocated":11202560,"data_stored":11202560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739745,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":31268,"kb_used_data":31268,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52266460,"statfs":{"total":53552873472,"available":53520855040,"internally_reserved":0,"allocated":32018432,"data_stored":32018432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53521772544,"internally_reserved":0,"allocated":31100928,"data_stored":31100928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53526876160,"internally_reserved":0,"allocated":25997312,"data_stored":25997312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T15:02:18.910 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T15:02:18.911 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T15:02:19.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:19.024+0000 7f2d5bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:19.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:19.033+0000 7f2d5bf91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:19.234 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T15:02:19.234 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-11T15:02:19.248 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":983,"stamp":"2024-09-11T15:02:18.897921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":62164,"kb_used_data":62164,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156831020,"statfs":{"total":160658620416,"available":160594964480,"internally_reserved":0,"allocated":63655936,"data_stored":63655936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5395,"num_objects":-57,"num_object_clones":0,"num_object_copies":-114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3695,"num_read_kb":-473,"num_write":-369,"num_write_kb":-234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.367654"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.639763+0000","last_change":"2024-09-11T14:39:10.591513+0000","last_active":"2024-09-11T15:02:15.639763+0000","last_peered":"2024-09-11T15:02:15.639763+0000","last_clean":"2024-09-11T15:02:15.639763+0000","last_became_active":"2024-09-11T14:39:08.649048+0000","last_became_peered":"2024-09-11T14:39:08.649048+0000","last_unstale":"2024-09-11T15:02:15.639763+0000","last_undegraded":"2024-09-11T15:02:15.639763+0000","last_fullsized":"2024-09-11T15:02:15.639763+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.581187+0000","last_change":"2024-09-11T14:39:10.590719+0000","last_active":"2024-09-11T15:02:15.581187+0000","last_peered":"2024-09-11T15:02:15.581187+0000","last_clean":"2024-09-11T15:02:15.581187+0000","last_became_active":"2024-09-11T14:39:08.671767+0000","last_became_peered":"2024-09-11T14:39:08.671767+0000","last_unstale":"2024-09-11T15:02:15.581187+0000","last_undegraded":"2024-09-11T15:02:15.581187+0000","last_fullsized":"2024-09-11T15:02:15.581187+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.612936+0000","last_change":"2024-09-11T14:39:10.595741+0000","last_active":"2024-09-11T15:02:15.612936+0000","last_peered":"2024-09-11T15:02:15.612936+0000","last_clean":"2024-09-11T15:02:15.612936+0000","last_became_active":"2024-09-11T14:39:08.728005+0000","last_became_peered":"2024-09-11T14:39:08.728005+0000","last_unstale":"2024-09-11T15:02:15.612936+0000","last_undegraded":"2024-09-11T15:02:15.612936+0000","last_fullsized":"2024-09-11T15:02:15.612936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.644307+0000","last_change":"2024-09-11T14:39:10.592063+0000","last_active":"2024-09-11T15:02:15.644307+0000","last_peered":"2024-09-11T15:02:15.644307+0000","last_clean":"2024-09-11T15:02:15.644307+0000","last_became_active":"2024-09-11T14:39:08.660946+0000","last_became_peered":"2024-09-11T14:39:08.660946+0000","last_unstale":"2024-09-11T15:02:15.644307+0000","last_undegraded":"2024-09-11T15:02:15.644307+0000","last_fullsized":"2024-09-11T15:02:15.644307+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":851,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.672831+0000","last_change":"2024-09-11T14:39:10.598829+0000","last_active":"2024-09-11T15:02:17.672831+0000","last_peered":"2024-09-11T15:02:17.672831+0000","last_clean":"2024-09-11T15:02:17.672831+0000","last_became_active":"2024-09-11T14:39:08.696312+0000","last_became_peered":"2024-09-11T14:39:08.696312+0000","last_unstale":"2024-09-11T15:02:17.672831+0000","last_undegraded":"2024-09-11T15:02:17.672831+0000","last_fullsized":"2024-09-11T15:02:17.672831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":843,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.665809+0000","last_change":"2024-09-11T14:39:10.595170+0000","last_active":"2024-09-11T15:02:15.665809+0000","last_peered":"2024-09-11T15:02:15.665809+0000","last_clean":"2024-09-11T15:02:15.665809+0000","last_became_active":"2024-09-11T14:39:08.722304+0000","last_became_peered":"2024-09-11T14:39:08.722304+0000","last_unstale":"2024-09-11T15:02:15.665809+0000","last_undegraded":"2024-09-11T15:02:15.665809+0000","last_fullsized":"2024-09-11T15:02:15.665809+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":843,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.636417+0000","last_change":"2024-09-11T14:39:10.588768+0000","last_active":"2024-09-11T15:02:15.636417+0000","last_peered":"2024-09-11T15:02:15.636417+0000","last_clean":"2024-09-11T15:02:15.636417+0000","last_became_active":"2024-09-11T14:39:08.670002+0000","last_became_peered":"2024-09-11T14:39:08.670002+0000","last_unstale":"2024-09-11T15:02:15.636417+0000","last_undegraded":"2024-09-11T15:02:15.636417+0000","last_fullsized":"2024-09-11T15:02:15.636417+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":842,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.633111+0000","last_change":"2024-09-11T14:39:10.596036+0000","last_active":"2024-09-11T15:02:15.633111+0000","last_peered":"2024-09-11T15:02:15.633111+0000","last_clean":"2024-09-11T15:02:15.633111+0000","last_became_active":"2024-09-11T14:39:08.669437+0000","last_became_peered":"2024-09-11T14:39:08.669437+0000","last_unstale":"2024-09-11T15:02:15.633111+0000","last_undegraded":"2024-09-11T15:02:15.633111+0000","last_fullsized":"2024-09-11T15:02:15.633111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":910,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-11T15:02:15.636984+0000","last_change":"2024-09-11T14:39:07.488096+0000","last_active":"2024-09-11T15:02:15.636984+0000","last_peered":"2024-09-11T15:02:15.636984+0000","last_clean":"2024-09-11T15:02:15.636984+0000","last_became_active":"2024-09-11T14:39:07.454105+0000","last_became_peered":"2024-09-11T14:39:07.454105+0000","last_unstale":"2024-09-11T15:02:15.636984+0000","last_undegraded":"2024-09-11T15:02:15.636984+0000","last_fullsized":"2024-09-11T15:02:15.636984+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_clean_scrub_stamp":"2024-09-11T14:39:06.320059+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160598142976,"internally_reserved":0,"allocated":60477440,"data_stored":60477440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654707043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":26560,"kb_used_data":26560,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52271168,"statfs":{"total":53552873472,"available":53525676032,"internally_reserved":0,"allocated":27197440,"data_stored":27197440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739750,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":10940,"kb_used_data":10940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52286788,"statfs":{"total":53552873472,"available":53541670912,"internally_reserved":0,"allocated":11202560,"data_stored":11202560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739750,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":24664,"kb_used_data":24664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52273064,"statfs":{"total":53552873472,"available":53527617536,"internally_reserved":0,"allocated":25255936,"data_stored":25255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53528850432,"internally_reserved":0,"allocated":24023040,"data_stored":24023040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53542416384,"internally_reserved":0,"allocated":10457088,"data_stored":10457088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53526876160,"internally_reserved":0,"allocated":25997312,"data_stored":25997312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T15:02:19.249 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T15:02:19.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:19.364+0000 7ffa708d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:19.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:19.373+0000 7ffa708d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:19.574 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T15:02:19.574 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":292,"fsid":"1af81b22-8a9d-4464-aa64-8b69c827c577","created":"2024-09-11T14:38:57.305066+0000","modified":"2024-09-11T15:02:17.515299+0000","last_up_change":"2024-09-11T14:39:05.311247+0000","last_in_change":"2024-09-11T14:38:59.230220+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:39:06.266706+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:39:07.179749+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38039c61-9ba6-47ac-ada1-6058644a16a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":823145234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":823145234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":823145234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":823145234}]},"public_addr":"172.21.15.59:6805/823145234","cluster_addr":"172.21.15.59:6804/823145234","heartbeat_back_addr":"172.21.15.59:6806/823145234","heartbeat_front_addr":"172.21.15.59:6809/823145234","state":["exists","up"]},{"osd":1,"uuid":"6c1cd617-bba7-42b4-8e50-d725dae9bf2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":1003425153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1003425153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":1003425153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":1003425153}]},"public_addr":"172.21.15.59:6803/1003425153","cluster_addr":"172.21.15.59:6802/1003425153","heartbeat_back_addr":"172.21.15.59:6808/1003425153","heartbeat_front_addr":"172.21.15.59:6807/1003425153","state":["exists","up"]},{"osd":2,"uuid":"3fa00c86-c849-4001-862e-9e53163090f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":242,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":746975144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":746975144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":746975144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":746975144}]},"public_addr":"172.21.15.59:6812/746975144","cluster_addr":"172.21.15.59:6811/746975144","heartbeat_back_addr":"172.21.15.59:6813/746975144","heartbeat_front_addr":"172.21.15.59:6810/746975144","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":7,"length":11},{"begin":23,"length":4},{"begin":37,"length":1},{"begin":41,"length":1},{"begin":44,"length":2},{"begin":69,"length":1},{"begin":71,"length":1}]},{"pool":4,"snaps":[{"begin":2,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":4}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":12,"snaps":[{"begin":2,"length":1}]},{"pool":13,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":2,"length":1}]},{"pool":16,"snaps":[{"begin":2,"length":1}]},{"pool":17,"snaps":[{"begin":2,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":2,"length":1}]},{"pool":20,"snaps":[{"begin":2,"length":1}]},{"pool":21,"snaps":[{"begin":2,"length":1}]},{"pool":22,"snaps":[{"begin":2,"length":3}]},{"pool":23,"snaps":[{"begin":2,"length":9}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T15:02:20.589 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T15:02:20.590 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T15:02:20.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:20.705+0000 7fc776111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:20.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:20.721+0000 7fc776111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:20.803 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-11T15:02:20.803 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "" 2024-09-11T15:02:20.803 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-11T15:02:20.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T15:02:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:20.932+0000 7fb959b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:20.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:20.943+0000 7fb959b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:21.153 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T15:02:21.169 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T15:02:21.169 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T15:02:21.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:21.287+0000 7f5fa9d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:21.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:21.301+0000 7f5fa9d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:21.320 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-11T15:02:21.320 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "" 2024-09-11T15:02:21.320 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-11T15:02:21.334 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T15:02:21.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:21.456+0000 7f69b3d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:21.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:21.466+0000 7f69b3d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:21.673 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T15:02:21.694 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T15:02:21.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T15:02:21.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:21.818+0000 7f4d79f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:21.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:21.833+0000 7f4d79f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:21.857 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-09-11T15:02:21.857 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "" 2024-09-11T15:02:21.857 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-09-11T15:02:21.870 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T15:02:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:21.985+0000 7ff2d7663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:21.996+0000 7ff2d7663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:22.197 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T15:02:22.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T15:02:22.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:22.331+0000 7ff380a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:22.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:22.341+0000 7ff380a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:22.540 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T15:02:22.541 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-11T15:02:22.555 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":984,"stamp":"2024-09-11T15:02:20.898272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":45408,"kb_used_data":45408,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156847776,"statfs":{"total":160658620416,"available":160612122624,"internally_reserved":0,"allocated":46497792,"data_stored":46497792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5395,"num_objects":-57,"num_object_clones":0,"num_object_copies":-114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3695,"num_read_kb":-473,"num_write":-369,"num_write_kb":-234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001082"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":847,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.580800+0000","last_change":"2024-09-11T14:39:10.591513+0000","last_active":"2024-09-11T15:02:17.580800+0000","last_peered":"2024-09-11T15:02:17.580800+0000","last_clean":"2024-09-11T15:02:17.580800+0000","last_became_active":"2024-09-11T14:39:08.649048+0000","last_became_peered":"2024-09-11T14:39:08.649048+0000","last_unstale":"2024-09-11T15:02:17.580800+0000","last_undegraded":"2024-09-11T15:02:17.580800+0000","last_fullsized":"2024-09-11T15:02:17.580800+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":847,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.588456+0000","last_change":"2024-09-11T14:39:10.590719+0000","last_active":"2024-09-11T15:02:17.588456+0000","last_peered":"2024-09-11T15:02:17.588456+0000","last_clean":"2024-09-11T15:02:17.588456+0000","last_became_active":"2024-09-11T14:39:08.671767+0000","last_became_peered":"2024-09-11T14:39:08.671767+0000","last_unstale":"2024-09-11T15:02:17.588456+0000","last_undegraded":"2024-09-11T15:02:17.588456+0000","last_fullsized":"2024-09-11T15:02:17.588456+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":847,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.656504+0000","last_change":"2024-09-11T14:39:10.595741+0000","last_active":"2024-09-11T15:02:17.656504+0000","last_peered":"2024-09-11T15:02:17.656504+0000","last_clean":"2024-09-11T15:02:17.656504+0000","last_became_active":"2024-09-11T14:39:08.728005+0000","last_became_peered":"2024-09-11T14:39:08.728005+0000","last_unstale":"2024-09-11T15:02:17.656504+0000","last_undegraded":"2024-09-11T15:02:17.656504+0000","last_fullsized":"2024-09-11T15:02:17.656504+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":847,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.621632+0000","last_change":"2024-09-11T14:39:10.592063+0000","last_active":"2024-09-11T15:02:17.621632+0000","last_peered":"2024-09-11T15:02:17.621632+0000","last_clean":"2024-09-11T15:02:17.621632+0000","last_became_active":"2024-09-11T14:39:08.660946+0000","last_became_peered":"2024-09-11T14:39:08.660946+0000","last_unstale":"2024-09-11T15:02:17.621632+0000","last_undegraded":"2024-09-11T15:02:17.621632+0000","last_fullsized":"2024-09-11T15:02:17.621632+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":851,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.672831+0000","last_change":"2024-09-11T14:39:10.598829+0000","last_active":"2024-09-11T15:02:17.672831+0000","last_peered":"2024-09-11T15:02:17.672831+0000","last_clean":"2024-09-11T15:02:17.672831+0000","last_became_active":"2024-09-11T14:39:08.696312+0000","last_became_peered":"2024-09-11T14:39:08.696312+0000","last_unstale":"2024-09-11T15:02:17.672831+0000","last_undegraded":"2024-09-11T15:02:17.672831+0000","last_fullsized":"2024-09-11T15:02:17.672831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":849,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.648563+0000","last_change":"2024-09-11T14:39:10.595170+0000","last_active":"2024-09-11T15:02:17.648563+0000","last_peered":"2024-09-11T15:02:17.648563+0000","last_clean":"2024-09-11T15:02:17.648563+0000","last_became_active":"2024-09-11T14:39:08.722304+0000","last_became_peered":"2024-09-11T14:39:08.722304+0000","last_unstale":"2024-09-11T15:02:17.648563+0000","last_undegraded":"2024-09-11T15:02:17.648563+0000","last_fullsized":"2024-09-11T15:02:17.648563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":849,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.671106+0000","last_change":"2024-09-11T14:39:10.588768+0000","last_active":"2024-09-11T15:02:17.671106+0000","last_peered":"2024-09-11T15:02:17.671106+0000","last_clean":"2024-09-11T15:02:17.671106+0000","last_became_active":"2024-09-11T14:39:08.670002+0000","last_became_peered":"2024-09-11T14:39:08.670002+0000","last_unstale":"2024-09-11T15:02:17.671106+0000","last_undegraded":"2024-09-11T15:02:17.671106+0000","last_fullsized":"2024-09-11T15:02:17.671106+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":848,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.660017+0000","last_change":"2024-09-11T14:39:10.596036+0000","last_active":"2024-09-11T15:02:17.660017+0000","last_peered":"2024-09-11T15:02:17.660017+0000","last_clean":"2024-09-11T15:02:17.660017+0000","last_became_active":"2024-09-11T14:39:08.669437+0000","last_became_peered":"2024-09-11T14:39:08.669437+0000","last_unstale":"2024-09-11T15:02:17.660017+0000","last_undegraded":"2024-09-11T15:02:17.660017+0000","last_fullsized":"2024-09-11T15:02:17.660017+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:07.323952+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":916,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:17.583661+0000","last_change":"2024-09-11T14:39:07.488096+0000","last_active":"2024-09-11T15:02:17.583661+0000","last_peered":"2024-09-11T15:02:17.583661+0000","last_clean":"2024-09-11T15:02:17.583661+0000","last_became_active":"2024-09-11T14:39:07.454105+0000","last_became_peered":"2024-09-11T14:39:07.454105+0000","last_unstale":"2024-09-11T15:02:17.583661+0000","last_undegraded":"2024-09-11T15:02:17.583661+0000","last_fullsized":"2024-09-11T15:02:17.583661+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:39:06.320059+0000","last_clean_scrub_stamp":"2024-09-11T14:39:06.320059+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549010944,"internally_reserved":0,"allocated":3862528,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160614879232,"internally_reserved":0,"allocated":43741184,"data_stored":43741184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654707048,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":16972,"kb_used_data":16972,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52280756,"statfs":{"total":53552873472,"available":53535494144,"internally_reserved":0,"allocated":17379328,"data_stored":17379328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739755,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3772,"kb_used_data":3772,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293956,"statfs":{"total":53552873472,"available":53549010944,"internally_reserved":0,"allocated":3862528,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739750,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":24664,"kb_used_data":24664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52273064,"statfs":{"total":53552873472,"available":53527617536,"internally_reserved":0,"allocated":25255936,"data_stored":25255936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53549010944,"internally_reserved":0,"allocated":3862528,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53528850432,"internally_reserved":0,"allocated":24023040,"data_stored":24023040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549010944,"internally_reserved":0,"allocated":3862528,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53537017856,"internally_reserved":0,"allocated":15855616,"data_stored":15855616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T15:02:22.560 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T14:39:07.323952+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=7, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.560 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T14:39:07.323952+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=7, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.560 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T14:39:07.323952+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=7, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.561 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T14:39:07.323952+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=7, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.561 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T14:39:07.323952+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=7, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.561 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T14:39:07.323952+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=7, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.561 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T14:39:07.323952+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=7, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.561 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T14:39:07.323952+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=7, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.561 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T14:39:06.320059+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=39, tm_sec=6, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=15, tm_min=2, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T15:02:22.562 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T15:02:42.563 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T15:02:42.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:42.685+0000 7fed43191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:42.693+0000 7fed43191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:42.890 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-09-11T15:02:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-09-11T15:02:42.902 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":994,"stamp":"2024-09-11T15:02:40.902676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":9884,"kb_used_data":9884,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156883300,"statfs":{"total":160658620416,"available":160648499200,"internally_reserved":0,"allocated":10121216,"data_stored":10121216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002762"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":849,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:21.680756+0000","last_change":"2024-09-11T15:02:21.733598+0000","last_active":"2024-09-11T15:02:21.680756+0000","last_peered":"2024-09-11T15:02:21.680756+0000","last_clean":"2024-09-11T15:02:21.680756+0000","last_became_active":"2024-09-11T14:39:08.649048+0000","last_became_peered":"2024-09-11T14:39:08.649048+0000","last_unstale":"2024-09-11T15:02:21.680756+0000","last_undegraded":"2024-09-11T15:02:21.680756+0000","last_fullsized":"2024-09-11T15:02:21.680756+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T15:02:21.733417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:21.733417+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":849,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:21.678941+0000","last_change":"2024-09-11T15:02:21.735865+0000","last_active":"2024-09-11T15:02:21.678941+0000","last_peered":"2024-09-11T15:02:21.678941+0000","last_clean":"2024-09-11T15:02:21.678941+0000","last_became_active":"2024-09-11T14:39:08.671767+0000","last_became_peered":"2024-09-11T14:39:08.671767+0000","last_unstale":"2024-09-11T15:02:21.678941+0000","last_undegraded":"2024-09-11T15:02:21.678941+0000","last_fullsized":"2024-09-11T15:02:21.678941+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T15:02:21.735668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:21.735668+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":849,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:21.678948+0000","last_change":"2024-09-11T15:02:21.715178+0000","last_active":"2024-09-11T15:02:21.678948+0000","last_peered":"2024-09-11T15:02:21.678948+0000","last_clean":"2024-09-11T15:02:21.678948+0000","last_became_active":"2024-09-11T14:39:08.728005+0000","last_became_peered":"2024-09-11T14:39:08.728005+0000","last_unstale":"2024-09-11T15:02:21.678948+0000","last_undegraded":"2024-09-11T15:02:21.678948+0000","last_fullsized":"2024-09-11T15:02:21.678948+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T15:02:21.714852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:21.714852+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":849,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:21.677534+0000","last_change":"2024-09-11T15:02:21.728580+0000","last_active":"2024-09-11T15:02:21.677534+0000","last_peered":"2024-09-11T15:02:21.677534+0000","last_clean":"2024-09-11T15:02:21.677534+0000","last_became_active":"2024-09-11T14:39:08.660946+0000","last_became_peered":"2024-09-11T14:39:08.660946+0000","last_unstale":"2024-09-11T15:02:21.677534+0000","last_undegraded":"2024-09-11T15:02:21.677534+0000","last_fullsized":"2024-09-11T15:02:21.677534+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T15:02:21.728397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:21.728397+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":853,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:21.159356+0000","last_change":"2024-09-11T15:02:21.195607+0000","last_active":"2024-09-11T15:02:21.159356+0000","last_peered":"2024-09-11T15:02:21.159356+0000","last_clean":"2024-09-11T15:02:21.159356+0000","last_became_active":"2024-09-11T14:39:08.696312+0000","last_became_peered":"2024-09-11T14:39:08.696312+0000","last_unstale":"2024-09-11T15:02:21.159356+0000","last_undegraded":"2024-09-11T15:02:21.159356+0000","last_fullsized":"2024-09-11T15:02:21.159356+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2024-09-11T15:02:21.195413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:21.195413+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":851,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:22.204964+0000","last_change":"2024-09-11T15:02:22.228906+0000","last_active":"2024-09-11T15:02:22.204964+0000","last_peered":"2024-09-11T15:02:22.204964+0000","last_clean":"2024-09-11T15:02:22.204964+0000","last_became_active":"2024-09-11T14:39:08.722304+0000","last_became_peered":"2024-09-11T14:39:08.722304+0000","last_unstale":"2024-09-11T15:02:22.204964+0000","last_undegraded":"2024-09-11T15:02:22.204964+0000","last_fullsized":"2024-09-11T15:02:22.204964+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T15:02:22.228717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:22.228717+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":851,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:22.203011+0000","last_change":"2024-09-11T15:02:22.238334+0000","last_active":"2024-09-11T15:02:22.203011+0000","last_peered":"2024-09-11T15:02:22.203011+0000","last_clean":"2024-09-11T15:02:22.203011+0000","last_became_active":"2024-09-11T14:39:08.670002+0000","last_became_peered":"2024-09-11T14:39:08.670002+0000","last_unstale":"2024-09-11T15:02:22.203011+0000","last_undegraded":"2024-09-11T15:02:22.203011+0000","last_fullsized":"2024-09-11T15:02:22.203011+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T15:02:22.238005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:22.238005+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":850,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:21.676863+0000","last_change":"2024-09-11T15:02:21.719923+0000","last_active":"2024-09-11T15:02:21.676863+0000","last_peered":"2024-09-11T15:02:21.676863+0000","last_clean":"2024-09-11T15:02:21.676863+0000","last_became_active":"2024-09-11T14:39:08.669437+0000","last_became_peered":"2024-09-11T14:39:08.669437+0000","last_unstale":"2024-09-11T15:02:21.676863+0000","last_undegraded":"2024-09-11T15:02:21.676863+0000","last_fullsized":"2024-09-11T15:02:21.676863+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2024-09-11T15:02:21.719564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:21.719564+0000","last_clean_scrub_stamp":"2024-09-11T14:39:07.323952+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":918,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-11T15:02:21.674946+0000","last_change":"2024-09-11T15:02:21.810592+0000","last_active":"2024-09-11T15:02:21.674946+0000","last_peered":"2024-09-11T15:02:21.674946+0000","last_clean":"2024-09-11T15:02:21.674946+0000","last_became_active":"2024-09-11T14:39:07.454105+0000","last_became_peered":"2024-09-11T14:39:07.454105+0000","last_unstale":"2024-09-11T15:02:21.674946+0000","last_undegraded":"2024-09-11T15:02:21.674946+0000","last_fullsized":"2024-09-11T15:02:21.674946+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-09-11T15:02:21.810370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T15:02:21.810370+0000","last_clean_scrub_stamp":"2024-09-11T14:39:06.320059+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549010944,"internally_reserved":0,"allocated":3862528,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160648499200,"internally_reserved":0,"allocated":10121216,"data_stored":10121216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654707068,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2596,"kb_used_data":2596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295132,"statfs":{"total":53552873472,"available":53550215168,"internally_reserved":0,"allocated":2658304,"data_stored":2658304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739775,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3772,"kb_used_data":3772,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293956,"statfs":{"total":53552873472,"available":53549010944,"internally_reserved":0,"allocated":3862528,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739770,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3516,"kb_used_data":3516,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294212,"statfs":{"total":53552873472,"available":53549273088,"internally_reserved":0,"allocated":3600384,"data_stored":3600384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53549010944,"internally_reserved":0,"allocated":3862528,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549273088,"internally_reserved":0,"allocated":3600384,"data_stored":3600384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549010944,"internally_reserved":0,"allocated":3862528,"data_stored":3862528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550215168,"internally_reserved":0,"allocated":2658304,"data_stored":2658304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T15:02:42.904 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T15:02:43.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:43.007+0000 7f7ce14fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-09-11T15:02:43.020+0000 7f7ce14fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T15:02:43.236 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T15:02:43.237 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T15:02:43.237 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T15:02:43.237 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T15:02:43.313 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T15:02:43.313 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T15:02:43.313 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T15:02:43.397 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T15:02:43.398 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T15:02:43.398 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T15:02:43.475 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T15:02:43.475 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T15:02:43.476 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T15:02:43.476 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T15:02:43.505 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T15:02:43.506 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T15:02:43.506 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T15:02:43.506 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T15:02:43.562 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T15:02:43.562 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T15:02:43.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(POOL_FULL\)' | head -n 1 2024-09-11T15:02:43.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T15:02:43.634 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T15:02:43.827 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T15:02:43.827 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T15:02:43.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T15:02:43.926 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T15:02:43.985 INFO:tasks.ceph:Archiving mon data... 2024-09-11T15:02:43.987 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901370/data/mon.a.tgz 2024-09-11T15:02:43.987 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-09-11T15:02:44.005 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.in6MduhqrH 2024-09-11T15:02:44.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.in6MduhqrH 2024-09-11T15:02:44.194 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.in6MduhqrH 2024-09-11T15:02:44.279 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.in6MduhqrH is 493KB 2024-09-11T15:02:44.377 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.in6MduhqrH 2024-09-11T15:02:44.393 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T15:02:44.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T15:02:44.513 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T15:02:44.513 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T15:02:44.515 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901370/remote/smithi059/crash 2024-09-11T15:02:44.515 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T15:02:44.578 INFO:tasks.ceph:Compressing logs... 2024-09-11T15:02:44.578 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T15:02:44.643 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T15:02:44.643 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39850.log 2024-09-11T15:02:44.657 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29996.log 2024-09-11T15:02:44.658 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.39850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28663.log 2024-09-11T15:02:44.658 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39850.log.gz 2024-09-11T15:02:44.658 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29736.log 2024-09-11T15:02:44.658 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.28663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29232.log 2024-09-11T15:02:44.658 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28663.log.gz 2024-09-11T15:02:44.659 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29148.log 2024-09-11T15:02:44.659 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29996.log.gz 2024-09-11T15:02:44.659 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29736.log.gz 2024-09-11T15:02:44.659 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29232.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T15:02:44.659 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29232.log.gz 2024-09-11T15:02:44.659 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40312.log 2024-09-11T15:02:44.659 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29148.log.gz 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28858.log 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28619.log 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40312.log.gz 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.28858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40060.log 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28858.log.gz 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.28619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29355.log 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28619.log.gz 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40060.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40060.log.gz 2024-09-11T15:02:44.660 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2024-09-11T15:02:44.661 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40144.log 2024-09-11T15:02:44.661 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29355.log.gz 2024-09-11T15:02:44.661 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40228.log 2024-09-11T15:02:44.661 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2024-09-11T15:02:44.661 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29607.log 2024-09-11T15:02:44.661 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40144.log.gz 2024-09-11T15:02:44.661 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40228.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T15:02:44.661 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40228.log.gz 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29053.log 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29607.log.gz 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28381.log 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29356.log 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29053.log.gz 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.28381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40352.log 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28381.log.gz 2024-09-11T15:02:44.662 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39892.log 2024-09-11T15:02:44.663 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29356.log.gz 2024-09-11T15:02:44.663 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29778.log 2024-09-11T15:02:44.663 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40352.log.gz 2024-09-11T15:02:44.663 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29190.log 2024-09-11T15:02:44.663 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39892.log.gz 2024-09-11T15:02:44.663 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28997.log 2024-09-11T15:02:44.663 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29778.log.gz 2024-09-11T15:02:44.663 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29482.log 2024-09-11T15:02:44.664 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29190.log.gz 2024-09-11T15:02:44.664 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.28997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40186.log 2024-09-11T15:02:44.664 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28997.log.gz 2024-09-11T15:02:44.664 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29482.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T15:02:44.665 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29482.log.gz 2024-09-11T15:02:44.665 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29442.log 2024-09-11T15:02:44.665 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40186.log.gz 2024-09-11T15:02:44.665 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29481.log 2024-09-11T15:02:44.666 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29442.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29442.log.gz 2024-09-11T15:02:44.666 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T15:02:44.666 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29274.log 2024-09-11T15:02:44.666 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29481.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29481.log.gz 2024-09-11T15:02:44.666 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29354.log 2024-09-11T15:02:44.666 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28828.log 2024-09-11T15:02:44.667 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29274.log.gz 2024-09-11T15:02:44.667 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-09-11T15:02:44.667 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29354.log.gz 2024-09-11T15:02:44.667 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.28828.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T15:02:44.667 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28828.log.gz 2024-09-11T15:02:44.667 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39934.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T15:02:44.667 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-09-11T15:02:44.668 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40018.log 2024-09-11T15:02:44.668 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39976.log 2024-09-11T15:02:44.668 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29568.log 2024-09-11T15:02:44.668 INFO:teuthology.orchestra.run.smithi059.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40018.log.gz 2024-09-11T15:02:44.668 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.39976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40270.log 2024-09-11T15:02:44.668 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39976.log.gz 2024-09-11T15:02:44.668 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28469.log 2024-09-11T15:02:44.668 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29568.log.gz 2024-09-11T15:02:44.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.40270.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40270.log.gz 2024-09-11T15:02:44.669 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29078.log 2024-09-11T15:02:44.669 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.28469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29608.log 2024-09-11T15:02:44.669 INFO:teuthology.orchestra.run.smithi059.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28469.log.gz 2024-09-11T15:02:44.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29694.log 2024-09-11T15:02:44.671 INFO:teuthology.orchestra.run.smithi059.stderr: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.29078.log.gz 2024-09-11T15:02:44.671 INFO:teuthology.orchestra.run.smithi059.stderr: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T15:02:44.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28579.log 2024-09-11T15:02:44.671 INFO:teuthology.orchestra.run.smithi059.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29608.log.gz 2024-09-11T15:02:44.671 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.29694.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29694.log.gz 2024-09-11T15:02:44.672 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph.tmp-client.admin.27428.log 2024-09-11T15:02:44.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.28579.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28579.log.gz 2024-09-11T15:02:44.672 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.tmp-client.admin.27428.log: 66.9% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27428.log.gz 2024-09-11T15:02:44.832 INFO:teuthology.orchestra.run.smithi059.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T15:02:45.271 INFO:teuthology.orchestra.run.smithi059.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T15:02:56.295 INFO:teuthology.orchestra.run.smithi059.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T15:02:57.426 INFO:teuthology.orchestra.run.smithi059.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T15:02:59.240 INFO:teuthology.orchestra.run.smithi059.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T15:02:59.242 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-09-11T15:02:59.242 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m14.608s 2024-09-11T15:02:59.242 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m38.319s 2024-09-11T15:02:59.242 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m1.472s 2024-09-11T15:02:59.242 INFO:tasks.ceph:Archiving logs... 2024-09-11T15:02:59.243 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901370/remote/smithi059/log 2024-09-11T15:02:59.244 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T15:03:04.523 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T15:03:04.533 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T15:03:04.534 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T15:03:04.596 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T15:03:04.604 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T15:03:04.605 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T15:03:04.627 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-09-11T15:03:04.631 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T15:03:04.631 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-09-11T15:03:04.631 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv01.front.sepia.ceph.com 3 6 377 7 +962us[ +971us] +/- 37ms 2024-09-11T15:03:04.632 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -912us[ -903us] +/- 67ms 2024-09-11T15:03:04.632 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -750us[ -750us] +/- 67ms 2024-09-11T15:03:04.632 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T15:03:04.633 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T15:03:04.641 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T15:03:04.642 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T15:03:04.649 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T15:03:04.743 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T15:03:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T15:03:04.773 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T15:03:04.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T15:03:04.788 INFO:teuthology.task.internal:Duration was 1853.619099 seconds 2024-09-11T15:03:04.789 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T15:03:04.796 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T15:03:04.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T15:03:04.855 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T15:03:05.180 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T15:03:05.180 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-09-11T15:03:05.180 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T15:03:05.244 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T15:03:05.244 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T15:03:05.315 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T15:03:05.315 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T15:03:05.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T15:03:05.472 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T15:03:05.472 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T15:03:05.498 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T15:03:05.507 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T15:03:05.562 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-09-11T15:03:05.622 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T15:03:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T15:03:05.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T15:03:05.687 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T15:03:05.688 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901370/remote/smithi059 2024-09-11T15:03:05.688 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T15:03:05.751 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T15:03:05.752 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T15:03:05.852 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T15:03:05.863 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T15:03:05.863 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T15:03:05.872 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T15:03:05.872 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T15:03:05.887 INFO:teuthology.orchestra.run.smithi059.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 11 15:03 /home/ubuntu/cephtest 2024-09-11T15:03:05.888 INFO:teuthology.orchestra.run.smithi059.stdout: 406098 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 11 14:39 /home/ubuntu/cephtest/mnt.0 2024-09-11T15:03:05.888 INFO:teuthology.orchestra.run.smithi059.stdout: 406099 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 11 14:40 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T15:03:05.888 INFO:teuthology.orchestra.run.smithi059.stdout: 420949 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 14:40 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T15:03:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T15:03:05.889 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-11T15:03:05.889 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-11T15:03:05.896 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T15:03:05.905 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T15:03:05.915 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi059 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-11T15:03:05.919 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests} duration: 1853.6190993785858 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi059 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6f077332b0e34d2e9264a30f779a20d2 status: fail success: false 2024-09-11T15:03:05.919 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T15:03:05.999 INFO:teuthology.run:FAIL