2024-08-16T02:07:59.187 INFO:root:teuthology version: 1.2.0 2024-08-16T02:07:59.202 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-16T02:07:59.285 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857403 branch: wip-yingxin-testing-2024-08-15-1551 description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7857403' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yingxin-testing-2024-08-15-1551 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(POOL_FULL\) sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 workunit: branch: wip-yingxin-testing-2024-08-15-1551 env: CRIMSON_COMPAT: '1' sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-yingxin-testing-2024-08-15-1551 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8Afig0gJfMSsbc7kZ7zOfui6Y7pV5+rk0SyAdceUYnPoLGg+3eDGQKvQWGp/aebAd93heT4x8fDO6qvxfgRhg= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-16_01:59:29 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-16T02:07:59.286 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa; will attempt to use it 2024-08-16T02:07:59.286 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks 2024-08-16T02:07:59.286 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-16T02:07:59.288 INFO:teuthology.task.internal:Checking packages... 2024-08-16T02:07:59.308 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '6f386a3f0b1fe6a4d7219b54bf963d790bc84004' 2024-08-16T02:07:59.308 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-16T02:07:59.308 INFO:teuthology.packaging:ref: None 2024-08-16T02:07:59.308 INFO:teuthology.packaging:tag: None 2024-08-16T02:07:59.308 INFO:teuthology.packaging:branch: wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:07:59.308 INFO:teuthology.packaging:sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:07:59.309 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:07:59.457 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5812.g6f386a3f 2024-08-16T02:07:59.458 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-16T02:07:59.464 INFO:teuthology.task.internal:no buildpackages task found 2024-08-16T02:07:59.464 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-16T02:07:59.510 INFO:teuthology.task.internal:Saving configuration 2024-08-16T02:07:59.521 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-16T02:07:59.543 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-16T02:07:59.568 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857403', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-16 02:02:50.246628', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8Afig0gJfMSsbc7kZ7zOfui6Y7pV5+rk0SyAdceUYnPoLGg+3eDGQKvQWGp/aebAd93heT4x8fDO6qvxfgRhg='} 2024-08-16T02:07:59.568 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-16T02:07:59.574 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-08-16T02:07:59.575 INFO:teuthology.run_tasks:Running task console_log... 2024-08-16T02:07:59.628 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3d57d9bac0>, signals=[15]) 2024-08-16T02:07:59.628 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-16T02:07:59.634 INFO:teuthology.task.internal:Opening connections... 2024-08-16T02:07:59.634 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:07:59.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:07:59.714 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-16T02:07:59.721 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-08-16T02:07:59.795 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-08-16T02:07:59.795 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-16T02:07:59.852 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-08-16T02:07:59.853 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-16T02:07:59.853 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-16T02:07:59.853 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-16T02:07:59.853 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-08-16T02:07:59.872 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-16T02:07:59.879 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-16T02:07:59.900 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-16T02:07:59.901 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-08-16T02:07:59.916 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-16T02:07:59.926 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-16T02:07:59.926 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-08-16T02:08:00.055 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-16T02:08:00.082 INFO:teuthology.run_tasks:Running task kernel... 2024-08-16T02:08:00.096 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-08-16T02:08:00.096 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-08-16T02:08:00.097 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-16T02:08:00.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:08:00.113 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-08-16T02:08:00.169 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-437.el9.x86_64 2024-08-16T02:08:00.169 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-437.el9.x86_64 2024-08-16T02:08:00.169 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-08-16T02:08:03.804 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.2 MB 00:02 2024-08-16T02:08:06.122 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 16 MB/s | 20 MB 00:01 2024-08-16T02:08:10.539 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-16T02:08:12.356 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 18 kB 00:00 2024-08-16T02:08:13.527 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-08-16T02:08:18.875 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-16T02:08:20.848 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-16T02:08:20.848 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-16T02:08:20.943 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-16T02:08:20.951 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-16T02:08:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:08:20.952 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:08:20.952 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:08:20.952 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-08-16T02:08:20.952 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:08:20.952 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 84 M 2024-08-16T02:08:20.952 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 126 M 2024-08-16T02:08:20.953 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:08:21.559 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 4.6 MB/s | 1.5 MB 00:00 2024-08-16T02:08:22.001 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 22 MB/s | 17 MB 00:00 2024-08-16T02:08:22.560 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 27 MB/s | 36 MB 00:01 2024-08-16T02:08:22.860 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 23 MB/s | 30 MB 00:01 2024-08-16T02:08:22.860 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:08:22.860 INFO:teuthology.orchestra.run.smithi087.stdout:Total 44 MB/s | 84 MB 00:01 2024-08-16T02:08:23.596 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:08:23.960 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:08:23.960 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:08:24.972 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:08:24.973 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:08:27.436 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:08:28.120 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:08:28.219 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:08:30.874 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:08:31.005 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:08:35.850 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:08:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:08:40.750 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:10.338 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:10.379 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:10.767 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:10.767 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:09:10.767 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:09:10.767 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:09:11.539 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:11.540 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:09:11.540 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:09:11.540 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-16T02:09:11.540 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-16T02:09:11.540 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-16T02:09:11.540 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-16T02:09:11.540 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:09:11.540 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:09:11.902 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:09:11.980 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:09:11.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:09:11.981 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-16T02:09:11.981 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-16T02:09:11.981 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-16T02:09:11.982 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-16T02:09:11.982 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-08-16T02:09:11.982 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:09:12.038 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:09:12.039 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:09:12.039 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2024-08-16T02:09:12.076 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.06 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 68.el9 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Group : Unspecified 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 8264254 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-16T02:09:12.077 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : builder@centos.org 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-16T02:09:12.078 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-16T02:09:12.079 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2024-08-16T02:09:12.079 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:09:12.079 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2024-08-16T02:09:12.079 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-16T02:09:12.080 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 5.14.0-480.el9.x86_64 2024-08-16T02:09:12.080 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-16T02:09:12.656 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-08-16T02:09:13.970 INFO:teuthology.orchestra.run.smithi087.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-16T02:09:14.000 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-08-16T02:09:14.008 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-08-16T02:09:14.036 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.Slq0cBKhf7 2024-08-16T02:09:14.037 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Slq0cBKhf7 2024-08-16T02:09:14.102 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.Slq0cBKhf7 2024-08-16T02:09:14.236 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.Slq0cBKhf7 is 6KB 2024-08-16T02:09:14.284 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.Slq0cBKhf7 2024-08-16T02:09:14.299 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2024-08-16T02:09:14.361 INFO:teuthology.orchestra.run.smithi087.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-16T02:09:14.361 INFO:teuthology.orchestra.run.smithi087.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-16T02:09:14.361 INFO:teuthology.orchestra.run.smithi087.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-16T02:09:14.361 INFO:teuthology.orchestra.run.smithi087.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-16T02:09:14.362 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-16T02:09:14.512 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-08-16T02:09:44.542 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-16T02:09:44.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-08-16T02:09:44.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:10:44.599 DEBUG:teuthology.orchestra.remote:timed out 2024-08-16T02:10:53.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-08-16T02:10:53.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:10:57.086 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-08-16T02:10:57.635 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-08-16T02:10:57.635 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-16T02:10:57.635 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-08-16T02:10:57.650 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-480.el9.x86_64 2024-08-16T02:10:57.650 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-16T02:10:57.650 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-16T02:10:57.650 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-16T02:10:58.652 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-16T02:10:58.652 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-16T02:10:58.733 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-08-16T02:10:58.768 DEBUG:teuthology.parallel:result is None 2024-08-16T02:10:58.769 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-16T02:10:58.776 INFO:teuthology.task.internal:Creating test directory... 2024-08-16T02:10:58.776 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-16T02:10:58.790 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-16T02:10:58.799 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-16T02:10:58.833 INFO:teuthology.task.internal:Creating archive directory... 2024-08-16T02:10:58.833 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-16T02:10:58.855 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-16T02:10:58.866 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-16T02:10:58.866 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-16T02:10:58.942 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:10:58.951 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:10:58.953 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-16T02:10:58.960 INFO:teuthology.task.internal:Configuring sudo... 2024-08-16T02:10:58.960 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-16T02:10:58.984 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-16T02:10:59.009 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-16T02:10:59.009 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:10:59.039 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:10:59.119 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:10:59.167 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:10:59.248 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:10:59.289 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:10:59.289 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-16T02:10:59.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-08-16T02:10:59.435 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:10:59.785 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-16T02:10:59.793 INFO:teuthology.task.internal:Starting timer... 2024-08-16T02:10:59.793 INFO:teuthology.run_tasks:Running task pcp... 2024-08-16T02:10:59.878 INFO:teuthology.run_tasks:Running task selinux... 2024-08-16T02:10:59.902 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-08-16T02:10:59.971 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-08-16T02:10:59.973 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-16T02:10:59.973 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-08-16T02:10:59.992 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-08-16T02:10:59.992 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive'} 2024-08-16T02:10:59.992 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:11:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:11:00.057 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-16T02:11:00.058 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-08-16T02:11:00.123 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-16T02:11:00.132 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-08-16T02:11:00.135 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-08-16T02:11:00.252 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-16T02:11:00.281 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-16T02:11:00.282 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-16T02:15:45.225 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com')] 2024-08-16T02:15:45.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-08-16T02:15:45.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:15:45.302 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-08-16T02:15:45.384 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-08-16T02:15:45.385 INFO:teuthology.run_tasks:Running task clock... 2024-08-16T02:15:45.406 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-16T02:15:45.406 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-16T02:15:45.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:15:45.454 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-16T02:15:45.468 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-16T02:15:45.492 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-08-16T02:15:45.503 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-08-16T02:15:45.517 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-16T02:15:45.531 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-16T02:15:45.588 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:15:45.591 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:15:45.591 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-08-16T02:15:45.592 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:15:45.592 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:15:45.592 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:15:45.592 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:15:45.593 INFO:teuthology.run_tasks:Running task install... 2024-08-16T02:15:45.602 DEBUG:teuthology.task.install:project ceph 2024-08-16T02:15:45.602 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}} 2024-08-16T02:15:45.602 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'} 2024-08-16T02:15:45.603 INFO:teuthology.task.install:Using flavor: crimson 2024-08-16T02:15:45.611 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-16T02:15:45.611 INFO:teuthology.task.install:extra packages: [] 2024-08-16T02:15:45.611 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'tag': None, 'wait_for_package': False} 2024-08-16T02:15:45.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:15:45.760 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/ 2024-08-16T02:15:45.760 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5812.g6f386a3f 2024-08-16T02:15:45.870 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-16T02:15:45.870 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:15:45.870 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-16T02:15:45.896 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-16T02:15:45.897 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-16T02:15:45.962 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-16T02:15:46.038 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-16T02:15:46.102 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-08-16T02:15:46.103 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-08-16T02:15:46.330 INFO:teuthology.orchestra.run.smithi087.stdout:45 files removed 2024-08-16T02:15:46.360 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-08-16T02:15:47.013 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 233 kB/s | 88 kB 00:00 2024-08-16T02:15:47.372 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 78 kB/s | 24 kB 00:00 2024-08-16T02:15:47.707 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 7.1 kB/s | 2.1 kB 00:00 2024-08-16T02:15:49.632 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-08-16T02:15:51.583 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-08-16T02:15:55.892 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-16T02:15:58.301 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 18 kB 00:01 2024-08-16T02:15:59.155 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 31 MB/s | 22 MB 00:00 2024-08-16T02:16:04.503 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-16T02:16:06.821 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 10 M 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.8 M 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.1 M 2024-08-16T02:16:06.832 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 20 M 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 25 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 807 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 558 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.4 M 2024-08-16T02:16:06.833 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 45 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 138 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 185 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 364 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 357 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 117 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-16T02:16:06.834 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout:Install 30 Packages 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-08-16T02:16:06.835 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:06.836 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 60 M 2024-08-16T02:16:06.836 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:16:07.688 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-base-19.0.0-5812.g6f386a3f.el9.x86 9.5 MB/s | 6.1 MB 00:00 2024-08-16T02:16:08.298 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-common-19.0.0-5812.g6f386a3f.el9.x 16 MB/s | 20 MB 00:01 2024-08-16T02:16:08.313 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): ceph-selinux-19.0.0-5812.g6f386a3f.el9. 40 kB/s | 25 kB 00:00 2024-08-16T02:16:08.413 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): libcephfs2-19.0.0-5812.g6f386a3f.el9.x8 6.8 MB/s | 807 kB 00:00 2024-08-16T02:16:08.447 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): libradosstriper1-19.0.0-5812.g6f386a3f. 4.1 MB/s | 558 kB 00:00 2024-08-16T02:16:08.572 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): ceph-radosgw-19.0.0-5812.g6f386a3f.el9. 6.8 MB/s | 10 MB 00:01 2024-08-16T02:16:08.614 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-ceph-argparse-19.0.0-5812.g6f38 270 kB/s | 45 kB 00:00 2024-08-16T02:16:08.681 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): librgw2-19.0.0-5812.g6f386a3f.el9.x86_6 20 MB/s | 5.4 MB 00:00 2024-08-16T02:16:08.698 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-ceph-common-19.0.0-5812.g6f386a 1.1 MB/s | 138 kB 00:00 2024-08-16T02:16:08.723 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): python3-cephfs-19.0.0-5812.g6f386a3f.e 1.7 MB/s | 185 kB 00:00 2024-08-16T02:16:08.757 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-rados-19.0.0-5812.g6f386a3f.el 4.7 MB/s | 364 kB 00:00 2024-08-16T02:16:08.802 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-rbd-19.0.0-5812.g6f386a3f.el9. 3.4 MB/s | 357 kB 00:00 2024-08-16T02:16:08.824 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): python3-rgw-19.0.0-5812.g6f386a3f.el9. 1.1 MB/s | 117 kB 00:00 2024-08-16T02:16:08.883 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 853 kB/s | 107 kB 00:00 2024-08-16T02:16:08.909 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 45 kB 00:00 2024-08-16T02:16:08.934 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-08-16T02:16:08.967 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-08-16T02:16:08.992 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 721 kB/s | 42 kB 00:00 2024-08-16T02:16:09.017 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-08-16T02:16:09.043 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-08-16T02:16:09.068 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 942 kB/s | 47 kB 00:00 2024-08-16T02:16:09.102 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-08-16T02:16:09.228 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-08-16T02:16:09.261 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.5 MB/s | 49 kB 00:00 2024-08-16T02:16:09.286 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 198 kB/s | 26 kB 00:00 2024-08-16T02:16:09.311 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-16T02:16:09.345 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-08-16T02:16:09.412 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-08-16T02:16:09.479 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-08-16T02:16:09.522 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-08-16T02:16:09.655 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): librados2-19.0.0-5812.g6f386a3f.el9.x8 14 MB/s | 3.5 MB 00:00 2024-08-16T02:16:09.722 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): librbd1-19.0.0-5812.g6f386a3f.el9.x86_ 16 MB/s | 3.8 MB 00:00 2024-08-16T02:16:09.726 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:16:09.726 INFO:teuthology.orchestra.run.smithi087.stdout:Total 21 MB/s | 60 MB 00:02 2024-08-16T02:16:09.825 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:16:09.861 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:16:09.861 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:16:10.438 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:16:10.439 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:16:11.884 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:16:12.412 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-16T02:16:12.465 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:16:12.545 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:16:12.642 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:16:12.667 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:16:12.713 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:16:12.756 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-08-16T02:16:12.790 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-08-16T02:16:12.822 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-08-16T02:16:12.845 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:16:12.870 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:16:13.055 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:16:13.086 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:16:13.446 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-08-16T02:16:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:16:13.574 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:16:13.618 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 12/34 2024-08-16T02:16:13.676 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:16:13.704 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 14/34 2024-08-16T02:16:13.783 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-08-16T02:16:13.810 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:16:13.900 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:16:13.934 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-08-16T02:16:14.041 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-08-16T02:16:14.076 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 19/34 2024-08-16T02:16:14.117 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-08-16T02:16:14.150 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-08-16T02:16:14.179 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 22/34 2024-08-16T02:16:14.201 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-08-16T02:16:14.238 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-16T02:16:14.846 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-16T02:16:14.875 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:16:14.948 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:16:15.025 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 27/34 2024-08-16T02:16:15.072 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 28/34 2024-08-16T02:16:17.488 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:16:17.530 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:16:18.238 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:16:18.275 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:16:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:16:18.352 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-16T02:16:18.352 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-16T02:16:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:16:28.248 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:16:28.248 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-08-16T02:16:28.249 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-08-16T02:16:28.249 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-08-16T02:16:28.249 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-08-16T02:16:28.249 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-08-16T02:16:28.249 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2024-08-16T02:16:28.249 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-08-16T02:16:28.249 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:29.363 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:16:29.388 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:16:29.389 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:16:29.389 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-16T02:16:29.389 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:16:29.389 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:16:29.389 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:29.390 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:16:29.474 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:16:29.474 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 6/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 7/34 2024-08-16T02:16:30.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 8/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 9/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 10/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 12/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-16T02:16:30.696 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-16T02:16:30.697 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-16T02:16:30.697 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-16T02:16:30.697 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-16T02:16:30.697 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-16T02:16:30.697 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-16T02:16:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-16T02:16:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-16T02:16:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-16T02:16:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:16:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-16T02:16:30.698 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/34 2024-08-16T02:16:31.617 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:16:31.617 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:31.617 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-08-16T02:16:31.617 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.617 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.617 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:16:31.617 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-16T02:16:31.617 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.618 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-16T02:16:31.619 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-16T02:16:31.620 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-16T02:16:31.620 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-3.el9.x86_64 2024-08-16T02:16:31.620 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-16T02:16:31.620 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:31.620 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:16:31.990 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-08-16T02:16:32.545 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:28 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:16:32.672 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:16:32.673 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:32.673 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:16:32.673 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:32.673 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:16:32.673 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 86 M 2024-08-16T02:16:32.673 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: libasan8 x86_64 13.3.1-2.1.el9 appstream 469 k 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 175 k 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:32.674 INFO:teuthology.orchestra.run.smithi087.stdout:Install 9 Packages 2024-08-16T02:16:32.675 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:32.675 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 88 M 2024-08-16T02:16:32.675 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 834 M 2024-08-16T02:16:32.675 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:16:33.294 INFO:teuthology.orchestra.run.smithi087.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 632 kB/s | 94 kB 00:00 2024-08-16T02:16:33.519 INFO:teuthology.orchestra.run.smithi087.stdout:(2/9): libcephsqlite-19.0.0-5812.g6f386a3f.el9. 468 kB/s | 175 kB 00:00 2024-08-16T02:16:33.561 INFO:teuthology.orchestra.run.smithi087.stdout:(3/9): libasan8-13.3.1-2.1.el9.x86_64.rpm 1.7 MB/s | 469 kB 00:00 2024-08-16T02:16:33.704 INFO:teuthology.orchestra.run.smithi087.stdout:(4/9): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-08-16T02:16:33.762 INFO:teuthology.orchestra.run.smithi087.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-08-16T02:16:33.788 INFO:teuthology.orchestra.run.smithi087.stdout:(6/9): libubsan-11.4.1-3.el9.x86_64.rpm 698 kB/s | 187 kB 00:00 2024-08-16T02:16:33.838 INFO:teuthology.orchestra.run.smithi087.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 846 kB/s | 64 kB 00:00 2024-08-16T02:16:33.880 INFO:teuthology.orchestra.run.smithi087.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.3 MB/s | 122 kB 00:00 2024-08-16T02:16:36.930 INFO:teuthology.orchestra.run.smithi087.stdout:(9/9): ceph-test-19.0.0-5812.g6f386a3f.el9.x86_ 23 MB/s | 86 MB 00:03 2024-08-16T02:16:36.933 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:16:36.933 INFO:teuthology.orchestra.run.smithi087.stdout:Total 21 MB/s | 88 MB 00:04 2024-08-16T02:16:37.016 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:16:37.038 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:16:37.039 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:16:37.645 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:16:37.645 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:16:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:16:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-08-16T02:16:38.655 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-08-16T02:16:38.701 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-08-16T02:16:38.727 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-08-16T02:16:38.758 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:16:38.783 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:16:38.862 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:16:38.910 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-08-16T02:16:38.935 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:16:50.512 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:16:50.600 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:16:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:16:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/9 2024-08-16T02:16:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/9 2024-08-16T02:16:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-08-16T02:16:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libasan8-13.3.1-2.1.el9.x86_64 4/9 2024-08-16T02:16:56.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:16:56.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-08-16T02:16:56.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-08-16T02:16:56.973 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-08-16T02:16:57.365 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-08-16T02:16:57.365 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:57.365 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:16:57.365 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:57.365 INFO:teuthology.orchestra.run.smithi087.stdout: libasan8-13.3.1-2.1.el9.x86_64 2024-08-16T02:16:57.366 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:57.366 INFO:teuthology.orchestra.run.smithi087.stdout: libubsan-11.4.1-3.el9.x86_64 2024-08-16T02:16:57.366 INFO:teuthology.orchestra.run.smithi087.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-08-16T02:16:57.366 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-16T02:16:57.366 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-16T02:16:57.366 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-16T02:16:57.367 INFO:teuthology.orchestra.run.smithi087.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-08-16T02:16:57.367 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:57.367 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:16:57.640 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-08-16T02:16:58.194 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:54 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:16:58.323 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:16:58.326 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:58.326 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:16:58.326 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:58.326 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:16:58.326 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.5 k 2024-08-16T02:16:58.326 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:16:58.326 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 2.2 M 2024-08-16T02:16:58.326 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 1.6 M 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 284 k 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.1 M 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 72 M 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-16T02:16:58.327 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-16T02:16:58.328 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout:Install 37 Packages 2024-08-16T02:16:58.329 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:16:58.330 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 86 M 2024-08-16T02:16:58.330 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 943 M 2024-08-16T02:16:58.330 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:16:59.503 INFO:teuthology.orchestra.run.smithi087.stdout:(1/37): ceph-19.0.0-5812.g6f386a3f.el9.x86_64.r 32 kB/s | 6.5 kB 00:00 2024-08-16T02:17:00.653 INFO:teuthology.orchestra.run.smithi087.stdout:(2/37): ceph-mds-19.0.0-5812.g6f386a3f.el9.x86_ 1.6 MB/s | 2.2 MB 00:01 2024-08-16T02:17:00.787 INFO:teuthology.orchestra.run.smithi087.stdout:(3/37): ceph-mgr-19.0.0-5812.g6f386a3f.el9.x86_ 1.1 MB/s | 1.6 MB 00:01 2024-08-16T02:17:00.903 INFO:teuthology.orchestra.run.smithi087.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5812.g6f38 2.4 MB/s | 284 kB 00:00 2024-08-16T02:17:01.112 INFO:teuthology.orchestra.run.smithi087.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-08-16T02:17:01.304 INFO:teuthology.orchestra.run.smithi087.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 6.4 MB/s | 1.2 MB 00:00 2024-08-16T02:17:01.388 INFO:teuthology.orchestra.run.smithi087.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2024-08-16T02:17:01.497 INFO:teuthology.orchestra.run.smithi087.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-08-16T02:17:01.588 INFO:teuthology.orchestra.run.smithi087.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.3 MB/s | 125 kB 00:00 2024-08-16T02:17:01.730 INFO:teuthology.orchestra.run.smithi087.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-08-16T02:17:02.080 INFO:teuthology.orchestra.run.smithi087.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 539 kB/s | 188 kB 00:00 2024-08-16T02:17:02.264 INFO:teuthology.orchestra.run.smithi087.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 940 kB/s | 172 kB 00:00 2024-08-16T02:17:02.389 INFO:teuthology.orchestra.run.smithi087.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 278 kB/s | 35 kB 00:00 2024-08-16T02:17:02.581 INFO:teuthology.orchestra.run.smithi087.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 403 kB/s | 77 kB 00:00 2024-08-16T02:17:02.715 INFO:teuthology.orchestra.run.smithi087.stdout:(15/37): ceph-mon-19.0.0-5812.g6f386a3f.el9.x86 1.6 MB/s | 5.1 MB 00:03 2024-08-16T02:17:02.740 INFO:teuthology.orchestra.run.smithi087.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 322 kB/s | 51 kB 00:00 2024-08-16T02:17:02.807 INFO:teuthology.orchestra.run.smithi087.stdout:(17/37): lua-devel-5.4.4-4.el9.x86_64.rpm 335 kB/s | 22 kB 00:00 2024-08-16T02:17:02.866 INFO:teuthology.orchestra.run.smithi087.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.6 MB/s | 151 kB 00:00 2024-08-16T02:17:02.882 INFO:teuthology.orchestra.run.smithi087.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 249 kB/s | 42 kB 00:00 2024-08-16T02:17:02.899 INFO:teuthology.orchestra.run.smithi087.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-08-16T02:17:02.958 INFO:teuthology.orchestra.run.smithi087.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-08-16T02:17:02.975 INFO:teuthology.orchestra.run.smithi087.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 759 kB/s | 11 kB 00:00 2024-08-16T02:17:02.992 INFO:teuthology.orchestra.run.smithi087.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-08-16T02:17:03.017 INFO:teuthology.orchestra.run.smithi087.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-08-16T02:17:03.043 INFO:teuthology.orchestra.run.smithi087.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 457 kB/s | 23 kB 00:00 2024-08-16T02:17:03.068 INFO:teuthology.orchestra.run.smithi087.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-08-16T02:17:03.094 INFO:teuthology.orchestra.run.smithi087.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-08-16T02:17:03.119 INFO:teuthology.orchestra.run.smithi087.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-08-16T02:17:03.144 INFO:teuthology.orchestra.run.smithi087.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-16T02:17:03.169 INFO:teuthology.orchestra.run.smithi087.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 665 kB/s | 16 kB 00:00 2024-08-16T02:17:03.194 INFO:teuthology.orchestra.run.smithi087.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-08-16T02:17:03.220 INFO:teuthology.orchestra.run.smithi087.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-08-16T02:17:03.245 INFO:teuthology.orchestra.run.smithi087.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-08-16T02:17:03.270 INFO:teuthology.orchestra.run.smithi087.stdout:(34/37): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-08-16T02:17:03.295 INFO:teuthology.orchestra.run.smithi087.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-08-16T02:17:03.320 INFO:teuthology.orchestra.run.smithi087.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-08-16T02:17:15.237 INFO:teuthology.orchestra.run.smithi087.stdout:(37/37): ceph-osd-19.0.0-5812.g6f386a3f.el9.x86 5.0 MB/s | 72 MB 00:14 2024-08-16T02:17:15.241 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:15.242 INFO:teuthology.orchestra.run.smithi087.stdout:Total 5.1 MB/s | 86 MB 00:16 2024-08-16T02:17:15.387 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:17:15.429 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:17:15.429 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:17:16.207 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:17:16.208 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:17:17.884 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:17:17.947 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-16T02:17:17.988 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-16T02:17:18.025 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-16T02:17:18.097 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-16T02:17:18.149 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-16T02:17:18.182 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-16T02:17:18.274 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-16T02:17:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-16T02:17:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-16T02:17:18.436 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-16T02:17:18.466 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-16T02:17:18.501 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-16T02:17:18.534 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-16T02:17:18.574 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-16T02:17:18.609 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-16T02:17:18.643 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-16T02:17:18.692 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-16T02:17:18.724 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-16T02:17:18.798 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-16T02:17:18.937 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-16T02:17:19.015 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-16T02:17:19.069 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-16T02:17:19.107 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-16T02:17:19.181 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-16T02:17:19.237 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-16T02:17:19.424 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-16T02:17:19.478 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-16T02:17:19.557 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-16T02:17:19.654 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-16T02:17:19.694 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-16T02:17:19.823 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-16T02:17:20.013 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 32/37 2024-08-16T02:17:20.037 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:17:20.062 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:17:20.063 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:20.063 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-16T02:17:20.063 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:17:20.063 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:17:20.063 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:31.130 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:17:31.158 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:17:31.159 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:31.159 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-16T02:17:31.159 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:17:31.159 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:17:31.159 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:31.690 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:17:31.713 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:17:31.713 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:31.713 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-16T02:17:31.713 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:17:31.713 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:17:31.713 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:31.940 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:17:31.963 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:17:31.963 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:31.964 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-16T02:17:31.964 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:17:31.964 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:17:31.964 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:32.028 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:17:38.817 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:17:38.817 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/37 2024-08-16T02:17:38.817 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/37 2024-08-16T02:17:38.817 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 6/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-16T02:17:38.818 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-16T02:17:38.819 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-16T02:17:38.821 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2024-08-16T02:17:39.761 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-16T02:17:39.762 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-16T02:17:39.763 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-16T02:17:39.764 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-16T02:17:39.764 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-16T02:17:39.764 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:39.764 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:17:40.131 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-08-16T02:17:40.685 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:36 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:17:40.770 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:17:40.814 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:17:40.815 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:17:40.815 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:17:40.877 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-08-16T02:17:41.421 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:37 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:17:41.548 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:17:41.549 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:41.549 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:17:41.549 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:41.549 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:17:41.549 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 773 k 2024-08-16T02:17:41.550 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:41.550 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:17:41.550 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:41.550 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-08-16T02:17:41.550 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:41.550 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 773 k 2024-08-16T02:17:41.550 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 779 k 2024-08-16T02:17:41.550 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:17:41.849 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-19.0.0-5812.g6f386a3f.el9.noarch.rpm 2.5 MB/s | 773 kB 00:00 2024-08-16T02:17:41.849 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:41.850 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.5 MB/s | 773 kB 00:00 2024-08-16T02:17:41.850 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:17:41.854 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:17:41.855 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:17:41.860 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:17:41.861 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:17:41.933 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:17:42.406 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:17:42.482 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:17:42.891 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:17:43.142 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:17:43.143 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:43.143 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:17:43.143 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:43.143 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:43.143 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:17:43.391 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-08-16T02:17:43.951 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:39 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:17:44.083 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:17:44.083 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:44.083 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-08-16T02:17:44.083 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:44.083 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 158 k 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 158 k 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 644 k 2024-08-16T02:17:44.084 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:17:44.309 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-19.0.0-5812.g6f386a 702 kB/s | 158 kB 00:00 2024-08-16T02:17:44.310 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:44.310 INFO:teuthology.orchestra.run.smithi087.stdout:Total 698 kB/s | 158 kB 00:00 2024-08-16T02:17:44.310 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:17:44.320 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:17:44.320 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:17:44.377 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:17:44.377 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:17:44.530 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:17:44.602 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:17:44.627 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:17:44.627 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:44.627 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-16T02:17:44.627 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:45.572 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:17:45.573 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:45.573 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:17:45.573 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:45.573 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:45.573 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:17:45.771 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-08-16T02:17:46.335 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:42 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:17:46.425 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:17:46.466 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:17:46.467 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:17:46.467 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:17:46.536 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-08-16T02:17:47.089 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:43 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:17:47.216 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:17:47.217 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:47.217 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:17:47.217 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 14 M 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 34 k 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 16 k 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-16T02:17:47.218 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-16T02:17:47.219 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-16T02:17:47.219 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:47.219 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:17:47.219 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:17:47.219 INFO:teuthology.orchestra.run.smithi087.stdout:Install 11 Packages 2024-08-16T02:17:47.219 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:47.219 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 18 M 2024-08-16T02:17:47.219 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 154 M 2024-08-16T02:17:47.220 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:17:47.604 INFO:teuthology.orchestra.run.smithi087.stdout:(1/11): ceph-prometheus-alerts-19.0.0-5812.g6f3 78 kB/s | 16 kB 00:00 2024-08-16T02:17:47.625 INFO:teuthology.orchestra.run.smithi087.stdout:(2/11): ceph-grafana-dashboards-19.0.0-5812.g6f 151 kB/s | 34 kB 00:00 2024-08-16T02:17:47.746 INFO:teuthology.orchestra.run.smithi087.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-08-16T02:17:47.813 INFO:teuthology.orchestra.run.smithi087.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-08-16T02:17:47.839 INFO:teuthology.orchestra.run.smithi087.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 772 kB/s | 19 kB 00:00 2024-08-16T02:17:47.872 INFO:teuthology.orchestra.run.smithi087.stdout:(6/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-08-16T02:17:47.931 INFO:teuthology.orchestra.run.smithi087.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-08-16T02:17:47.948 INFO:teuthology.orchestra.run.smithi087.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-08-16T02:17:47.973 INFO:teuthology.orchestra.run.smithi087.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-08-16T02:17:48.023 INFO:teuthology.orchestra.run.smithi087.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-08-16T02:17:48.240 INFO:teuthology.orchestra.run.smithi087.stdout:(11/11): ceph-mgr-dashboard-19.0.0-5812.g6f386a 17 MB/s | 14 MB 00:00 2024-08-16T02:17:48.244 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:48.244 INFO:teuthology.orchestra.run.smithi087.stdout:Total 18 MB/s | 18 MB 00:01 2024-08-16T02:17:48.274 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:17:48.293 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:17:48.293 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:17:48.526 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:17:48.526 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:17:49.076 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:17:49.154 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-08-16T02:17:49.185 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-08-16T02:17:49.284 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-08-16T02:17:49.419 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-08-16T02:17:49.480 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-08-16T02:17:49.563 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-08-16T02:17:49.615 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-08-16T02:17:49.644 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:17:49.676 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 9/11 2024-08-16T02:17:52.830 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 10/11 2024-08-16T02:17:52.880 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 1/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 2/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 3/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:17:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-08-16T02:17:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-08-16T02:17:55.026 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-08-16T02:17:55.026 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:55.026 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:17:55.026 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-16T02:17:55.026 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:55.026 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:55.027 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:17:55.283 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-16T02:17:55.826 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:51 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:17:55.954 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:17:55.955 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-08-16T02:17:55.955 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:17:55.955 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 7.4 M 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-16T02:17:55.956 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-16T02:17:55.957 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-16T02:17:55.957 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-16T02:17:55.957 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:55.957 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:17:55.957 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-08-16T02:17:55.957 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-08-16T02:17:55.957 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:17:55.958 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 38 M 2024-08-16T02:17:55.958 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 209 M 2024-08-16T02:17:55.958 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:17:56.487 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.7 MB/s | 803 kB 00:00 2024-08-16T02:17:56.513 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-08-16T02:17:56.671 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 178 kB/s | 33 kB 00:00 2024-08-16T02:17:56.755 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 217 kB/s | 18 kB 00:00 2024-08-16T02:17:56.839 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 423 kB/s | 35 kB 00:00 2024-08-16T02:17:56.939 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-58 13 MB/s | 7.4 MB 00:00 2024-08-16T02:17:57.123 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-08-16T02:17:57.239 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 813 kB/s | 244 kB 00:00 2024-08-16T02:17:57.315 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2024-08-16T02:17:57.340 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-08-16T02:17:57.448 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-08-16T02:17:57.724 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-08-16T02:17:57.724 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:57.725 INFO:teuthology.orchestra.run.smithi087.stdout:Total 22 MB/s | 38 MB 00:01 2024-08-16T02:17:58.075 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:17:58.102 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:17:58.102 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:17:58.541 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:17:58.541 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:17:59.305 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:17:59.406 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-16T02:17:59.447 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:17:59.546 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:17:59.590 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-16T02:17:59.768 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-16T02:17:59.797 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-16T02:17:59.865 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-16T02:18:00.518 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-16T02:18:00.643 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-16T02:18:01.890 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:18:02.928 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-16T02:18:02.962 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:18:05.503 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 1/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:18:05.504 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-16T02:18:05.902 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-16T02:18:05.903 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-16T02:18:05.903 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-16T02:18:05.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-16T02:18:05.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-16T02:18:05.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-16T02:18:05.903 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-16T02:18:05.903 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:05.903 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:06.280 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-08-16T02:18:06.841 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:02 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:06.969 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:06.970 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 49 k 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-16T02:18:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-08-16T02:18:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:06.973 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.2 M 2024-08-16T02:18:06.973 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2024-08-16T02:18:06.973 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:07.691 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 288 kB/s | 26 kB 00:00 2024-08-16T02:18:07.715 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 168 kB/s | 19 kB 00:00 2024-08-16T02:18:07.791 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-08-16T02:18:07.816 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-08-16T02:18:07.841 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): ceph-mgr-rook-19.0.0-5812.g6f386a3f.el9 205 kB/s | 49 kB 00:00 2024-08-16T02:18:07.866 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-08-16T02:18:07.892 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-08-16T02:18:07.992 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-08-16T02:18:08.017 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 94 kB/s | 14 kB 00:00 2024-08-16T02:18:08.043 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-16T02:18:08.068 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-google-auth-2.33.0-1.el9.noarc 1.7 MB/s | 220 kB 00:00 2024-08-16T02:18:08.094 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-08-16T02:18:08.135 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-08-16T02:18:08.138 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:08.138 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-08-16T02:18:08.200 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:08.214 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:08.215 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:08.303 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:08.303 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:08.809 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:08.974 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-16T02:18:09.029 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-16T02:18:09.093 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-16T02:18:09.130 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-16T02:18:09.172 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-16T02:18:09.282 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-16T02:18:09.383 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-16T02:18:09.432 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-16T02:18:10.159 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-16T02:18:10.204 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-16T02:18:10.237 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-16T02:18:10.287 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-16T02:18:10.309 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:18:11.076 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:18:11.076 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 1/13 2024-08-16T02:18:11.076 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-16T02:18:11.076 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-16T02:18:11.076 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-16T02:18:11.076 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-16T02:18:11.077 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-16T02:18:11.077 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-16T02:18:11.077 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-16T02:18:11.077 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-16T02:18:11.077 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-16T02:18:11.077 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-16T02:18:11.077 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-16T02:18:11.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-16T02:18:11.415 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-16T02:18:11.415 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-16T02:18:11.415 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-16T02:18:11.415 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-16T02:18:11.415 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-16T02:18:11.415 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-16T02:18:11.415 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:11.415 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:11.654 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-08-16T02:18:12.202 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:08 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:12.330 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:12.331 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:12.331 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:18:12.331 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:12.331 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:12.331 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 150 k 2024-08-16T02:18:12.331 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-08-16T02:18:12.332 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:12.333 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 7.0 M 2024-08-16T02:18:12.333 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 33 M 2024-08-16T02:18:12.333 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:12.716 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-08-16T02:18:12.791 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-08-16T02:18:12.816 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): ceph-mgr-cephadm-19.0.0-5812.g6f386a3f.e 518 kB/s | 150 kB 00:00 2024-08-16T02:18:12.841 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-08-16T02:18:12.900 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 891 kB/s | 74 kB 00:00 2024-08-16T02:18:12.984 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-08-16T02:18:12.986 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:12.986 INFO:teuthology.orchestra.run.smithi087.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-08-16T02:18:13.056 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:13.067 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:13.067 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:13.163 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:13.164 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:13.631 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-16T02:18:13.685 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-16T02:18:14.211 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-16T02:18:14.296 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-16T02:18:14.362 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-16T02:18:14.391 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:18:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:18:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/6 2024-08-16T02:18:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-16T02:18:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-16T02:18:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-16T02:18:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-16T02:18:15.398 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:15.399 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:15.616 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-08-16T02:18:16.172 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:12 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:16.305 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:16.305 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:16.305 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 883 k 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 963 k 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 4.1 M 2024-08-16T02:18:16.306 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:16.711 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): ceph-fuse-19.0.0-5812.g6f386a3f.el9.x86_ 3.0 MB/s | 883 kB 00:00 2024-08-16T02:18:16.853 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 184 kB/s | 80 kB 00:00 2024-08-16T02:18:16.854 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:16.854 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.7 MB/s | 963 kB 00:00 2024-08-16T02:18:16.866 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:16.890 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:16.890 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:16.968 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:16.968 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:17.149 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:17.300 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-16T02:18:17.379 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:18:18.043 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:18:18.043 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/2 2024-08-16T02:18:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-16T02:18:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:18.323 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-16T02:18:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:18.323 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:18.569 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2024-08-16T02:18:19.125 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:15 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:19.259 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:19.260 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:19.260 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-08-16T02:18:19.260 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:19.260 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:19.260 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 285 k 2024-08-16T02:18:19.261 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:19.261 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:19.261 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:19.261 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-08-16T02:18:19.261 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:19.261 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 285 k 2024-08-16T02:18:19.261 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.3 M 2024-08-16T02:18:19.261 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:19.529 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-volume-19.0.0-5812.g6f386a3f.el9.noarch.rp 1.0 MB/s | 285 kB 00:00 2024-08-16T02:18:19.530 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:19.530 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.0 MB/s | 285 kB 00:00 2024-08-16T02:18:19.530 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:19.538 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:19.538 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:19.567 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:19.567 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:19.782 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:19.856 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:18:19.872 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:18:19.872 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:18:19.872 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-16T02:18:19.872 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:18:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:20.881 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:18:20.881 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:20.881 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:21.182 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-08-16T02:18:21.742 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:17 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:21.877 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 125 k 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:21.878 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 125 k 2024-08-16T02:18:21.879 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 445 k 2024-08-16T02:18:21.879 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:22.109 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-19.0.0-5812.g6f386a3f.el9.x86_64 542 kB/s | 125 kB 00:00 2024-08-16T02:18:22.110 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:22.110 INFO:teuthology.orchestra.run.smithi087.stdout:Total 538 kB/s | 125 kB 00:00 2024-08-16T02:18:22.110 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:22.116 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:22.116 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:22.179 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:22.180 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:22.343 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:22.449 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:22.928 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:23.182 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:23.182 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:23.182 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:23.182 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:23.182 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:23.182 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:23.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-08-16T02:18:24.038 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:20 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:24.127 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:24.173 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:24.174 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:18:24.174 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:24.233 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-08-16T02:18:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:20 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:24.908 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:24.909 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:24.909 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-08-16T02:18:24.909 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 32 k 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 32 k 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 142 k 2024-08-16T02:18:24.910 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:25.091 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-19.0.0-5812.g6f386a3f.el9.x86_6 176 kB/s | 32 kB 00:00 2024-08-16T02:18:25.091 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:25.091 INFO:teuthology.orchestra.run.smithi087.stdout:Total 174 kB/s | 32 kB 00:00 2024-08-16T02:18:25.091 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:25.095 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:25.095 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:25.112 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:25.113 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:25.195 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:25.295 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:25.687 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:25.913 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:25.913 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:25.913 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:25.913 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:25.913 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:25.913 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:26.082 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-08-16T02:18:26.637 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:22 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:26.725 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:26.768 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:26.769 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:18:26.769 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:26.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-08-16T02:18:27.386 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:23 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:27.474 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:27.518 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:27.519 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:18:27.519 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:27.574 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-08-16T02:18:28.125 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:24 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:28.210 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:28.252 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:28.253 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:18:28.253 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:28.322 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-08-16T02:18:28.882 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:24 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:28.967 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:29.010 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:29.011 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:18:29.011 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:29.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-08-16T02:18:29.623 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:25 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:29.708 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:29.750 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:29.751 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:18:29.751 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:29.821 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-08-16T02:18:30.371 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:26 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:30.457 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:30.499 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:30.500 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-08-16T02:18:30.500 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:30.559 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-08-16T02:18:31.112 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:27 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:31.244 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:31.244 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:31.244 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 91 k 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 91 k 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 321 k 2024-08-16T02:18:31.245 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:31.469 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 407 kB/s | 91 kB 00:00 2024-08-16T02:18:31.469 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:31.470 INFO:teuthology.orchestra.run.smithi087.stdout:Total 404 kB/s | 91 kB 00:00 2024-08-16T02:18:31.470 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:31.479 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:31.479 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:31.542 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:31.542 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:31.686 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:31.783 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:32.130 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:32.386 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:32.386 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:32.386 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:32.386 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:32.386 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:32.387 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:32.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-08-16T02:18:33.202 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:29 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:33.335 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:33.335 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:33.335 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-08-16T02:18:33.335 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:33.335 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:33.335 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:18:33.335 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:33.335 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:33.336 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:33.336 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-08-16T02:18:33.336 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:33.336 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.5 M 2024-08-16T02:18:33.336 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 30 M 2024-08-16T02:18:33.336 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:33.716 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 9.2 MB/s | 3.5 MB 00:00 2024-08-16T02:18:33.716 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:33.717 INFO:teuthology.orchestra.run.smithi087.stdout:Total 9.2 MB/s | 3.5 MB 00:00 2024-08-16T02:18:33.717 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:33.728 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:33.728 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:33.812 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:33.813 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:34.394 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:34.476 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:18:34.476 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-16T02:18:34.476 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:18:34.476 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:18:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:35.627 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:35.627 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:35.627 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:35.627 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:35.627 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:35.628 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:35.850 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-08-16T02:18:36.417 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:32 ago on Fri 16 Aug 2024 02:16:04 AM UTC. 2024-08-16T02:18:36.552 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-08-16T02:18:36.553 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:36.553 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-08-16T02:18:36.553 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:36.553 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-08-16T02:18:36.553 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 177 k 2024-08-16T02:18:36.553 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:36.554 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-08-16T02:18:36.554 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-08-16T02:18:36.554 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-08-16T02:18:36.554 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:36.554 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 177 k 2024-08-16T02:18:36.554 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 685 k 2024-08-16T02:18:36.554 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-08-16T02:18:36.777 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 793 kB/s | 177 kB 00:00 2024-08-16T02:18:36.778 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:36.778 INFO:teuthology.orchestra.run.smithi087.stdout:Total 787 kB/s | 177 kB 00:00 2024-08-16T02:18:36.778 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-08-16T02:18:36.788 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-08-16T02:18:36.788 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-08-16T02:18:36.851 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-08-16T02:18:36.851 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-08-16T02:18:37.021 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-08-16T02:18:37.117 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:37.546 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:37.803 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:37.803 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:37.803 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-08-16T02:18:37.803 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:37.803 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:37.803 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-08-16T02:18:38.043 DEBUG:teuthology.parallel:result is None 2024-08-16T02:18:38.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:18:38.191 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-16T02:18:38.213 INFO:teuthology.orchestra.run.smithi087.stdout:19.0.0-5812.g6f386a3f.el9 2024-08-16T02:18:38.214 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5812.g6f386a3f.el9 2024-08-16T02:18:38.214 INFO:teuthology.task.install:The correct ceph version 19.0.0-5812.g6f386a3f is installed. 2024-08-16T02:18:38.215 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-16T02:18:38.216 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:38.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-16T02:18:38.287 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-16T02:18:38.288 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:38.288 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-08-16T02:18:38.406 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-16T02:18:38.472 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-16T02:18:38.472 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:38.472 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-16T02:18:38.578 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-16T02:18:38.644 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-16T02:18:38.644 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:38.644 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-08-16T02:18:38.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-16T02:18:38.797 INFO:teuthology.run_tasks:Running task ceph... 2024-08-16T02:18:39.068 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-16T02:18:39.068 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2024-08-16T02:18:39.094 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-16T02:18:39.095 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-16T02:18:39.160 INFO:tasks.ceph:Creating extra log directories... 2024-08-16T02:18:39.160 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-16T02:18:39.233 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-16T02:18:39.233 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-16T02:18:39.234 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857403', 'branch': 'wip-yingxin-testing-2024-08-15-1551', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7857403', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-testing-2024-08-15-1551'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}}, 'workunit': {'branch': 'wip-yingxin-testing-2024-08-15-1551', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-yingxin-testing-2024-08-15-1551', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'targets': {'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8Afig0gJfMSsbc7kZ7zOfui6Y7pV5+rk0SyAdceUYnPoLGg+3eDGQKvQWGp/aebAd93heT4x8fDO6qvxfgRhg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '67db276914d29505c9707d04f3a3b8fb9d8aee4b', 'timestamp': '2024-08-16_01:59:29', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-16T02:18:39.234 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-16T02:18:39.282 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2024-08-16T02:18:39.347 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:39.347 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-08-16T02:18:39.403 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-16T02:18:39.403 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-08-16T02:18:39.459 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-16T02:18:39.460 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:18:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 676 Links: 1 2024-08-16T02:18:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:18:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:18:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-08-16 02:18:34.735060239 +0000 2024-08-16T02:18:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-08-16 02:14:53.615454485 +0000 2024-08-16T02:18:39.460 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-08-16 02:14:53.615454485 +0000 2024-08-16T02:18:39.460 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-08-16 02:14:53.615454485 +0000 2024-08-16T02:18:39.460 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-16T02:18:39.523 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-08-16T02:18:39.523 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-08-16T02:18:39.523 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000203265 s, 2.5 MB/s 2024-08-16T02:18:39.525 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-16T02:18:39.581 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-08-16T02:18:39.637 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-16T02:18:39.637 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:18:39.637 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 694 Links: 1 2024-08-16T02:18:39.637 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:18:39.637 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:18:39.637 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-08-16 02:18:34.737060195 +0000 2024-08-16T02:18:39.638 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-08-16 02:14:53.953446822 +0000 2024-08-16T02:18:39.638 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-08-16 02:14:53.953446822 +0000 2024-08-16T02:18:39.638 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-08-16 02:14:53.953446822 +0000 2024-08-16T02:18:39.638 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-16T02:18:39.701 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-08-16T02:18:39.701 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-08-16T02:18:39.701 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000204598 s, 2.5 MB/s 2024-08-16T02:18:39.702 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-16T02:18:39.759 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 713 Links: 1 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-08-16 02:18:34.737060195 +0000 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-08-16 02:14:54.272439590 +0000 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-08-16 02:14:54.272439590 +0000 2024-08-16T02:18:39.814 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-08-16 02:14:54.272439590 +0000 2024-08-16T02:18:39.815 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-16T02:18:39.876 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-08-16T02:18:39.876 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-08-16T02:18:39.876 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000207007 s, 2.5 MB/s 2024-08-16T02:18:39.877 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-16T02:18:39.933 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-08-16T02:18:39.988 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-16T02:18:39.989 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:18:39.989 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 732 Links: 1 2024-08-16T02:18:39.989 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:18:39.989 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:18:39.989 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-08-16 02:18:34.737060195 +0000 2024-08-16T02:18:39.989 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-08-16 02:14:54.590432380 +0000 2024-08-16T02:18:39.989 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-08-16 02:14:54.590432380 +0000 2024-08-16T02:18:39.989 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-08-16 02:14:54.590432380 +0000 2024-08-16T02:18:39.989 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-16T02:18:40.051 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-08-16T02:18:40.052 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-08-16T02:18:40.052 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000215847 s, 2.4 MB/s 2024-08-16T02:18:40.053 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-16T02:18:40.109 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-16T02:18:40.109 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-16T02:18:40.109 INFO:tasks.ceph:Generating config... 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-16T02:18:40.112 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-08-16T02:18:40.113 INFO:tasks.ceph:[osd] debug monc = 20 2024-08-16T02:18:40.113 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-08-16T02:18:40.113 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-08-16T02:18:40.113 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-16T02:18:40.113 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-16T02:18:40.113 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-08-16T02:18:40.113 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-08-16T02:18:40.113 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:18:40.113 INFO:tasks.ceph:[global] ms client mode = crc 2024-08-16T02:18:40.113 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-08-16T02:18:40.113 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-08-16T02:18:40.113 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-08-16T02:18:40.113 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-08-16T02:18:40.113 INFO:tasks.ceph:[global] ms service mode = crc 2024-08-16T02:18:40.113 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-16T02:18:40.114 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-16T02:18:40.114 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-08-16T02:18:40.114 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-08-16T02:18:40.114 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-08-16T02:18:40.114 INFO:tasks.ceph:Setting up mon.a... 2024-08-16T02:18:40.114 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-16T02:18:40.210 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.keyring 2024-08-16T02:18:40.213 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-16T02:18:40.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:18:40.288 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.87')] 2024-08-16T02:18:40.289 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.87', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-08-16T02:18:40.289 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:40.289 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-16T02:18:40.346 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.87 --print /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:40.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:40.436+0000 7fb2cdcfaf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:40.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:40.436+0000 7fb2cdcfaf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:40.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:40.436+0000 7fb2cdcfaf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:40.438 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:40.438 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: generated fsid a2704e77-df6d-4e59-a0c8-3375fbc7352b 2024-08-16T02:18:40.438 INFO:teuthology.orchestra.run.smithi087.stdout:setting min_mon_release = quincy 2024-08-16T02:18:40.438 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 0 2024-08-16T02:18:40.439 INFO:teuthology.orchestra.run.smithi087.stdout:fsid a2704e77-df6d-4e59-a0c8-3375fbc7352b 2024-08-16T02:18:40.439 INFO:teuthology.orchestra.run.smithi087.stdout:last_changed 2024-08-16T02:18:40.438146+0000 2024-08-16T02:18:40.439 INFO:teuthology.orchestra.run.smithi087.stdout:created 2024-08-16T02:18:40.438146+0000 2024-08-16T02:18:40.439 INFO:teuthology.orchestra.run.smithi087.stdout:min_mon_release 17 (quincy) 2024-08-16T02:18:40.439 INFO:teuthology.orchestra.run.smithi087.stdout:election_strategy: 1 2024-08-16T02:18:40.439 INFO:teuthology.orchestra.run.smithi087.stdout:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-08-16T02:18:40.439 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-08-16T02:18:40.440 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-16T02:18:40.455 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a2704e77-df6d-4e59-a0c8-3375fbc7352b... 2024-08-16T02:18:40.456 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-16T02:18:40.538 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-08-16T02:18:40.538 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2024-08-16T02:18:40.538 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-16T02:18:40.538 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2024-08-16T02:18:40.538 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.539 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-16T02:18:40.540 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = 172.21.15.87 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:18:40.541 INFO:teuthology.orchestra.run.smithi087.stdout: ms client mode = crc 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: ms cluster mode = crc 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: ms mon client mode = crc 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: ms mon cluster mode = crc 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: ms mon service mode = crc 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: ms service mode = crc 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = a2704e77-df6d-4e59-a0c8-3375fbc7352b 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2024-08-16T02:18:40.542 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.543 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: debug monc = 20 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: crimson alien thread cpu cores = 6-7 2024-08-16T02:18:40.544 INFO:teuthology.orchestra.run.smithi087.stdout: crimson osd obc lru size = 10 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = seastore 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: osd shutdown pgref assert = True 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2024-08-16T02:18:40.545 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.546 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: mon min osdmap epochs = 50 2024-08-16T02:18:40.547 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune interval = 2 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune min = 15 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune txsize = 2 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: paxos service trim min = 10 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default crimson = True 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2024-08-16T02:18:40.548 INFO:teuthology.orchestra.run.smithi087.stdout: crimson seastar cpu cores = 0-1 2024-08-16T02:18:40.549 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2024-08-16T02:18:40.549 INFO:teuthology.orchestra.run.smithi087.stdout: crimson seastar cpu cores = 2-3 2024-08-16T02:18:40.549 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2024-08-16T02:18:40.549 INFO:teuthology.orchestra.run.smithi087.stdout: crimson seastar cpu cores = 4-5 2024-08-16T02:18:40.551 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-16T02:18:40.555 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-16T02:18:40.654 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-16T02:18:40.654 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:40.654 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-16T02:18:40.670 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:40.670 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-16T02:18:40.725 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:18:40.725 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:40.725 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-16T02:18:40.725 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:18:40.799 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:40.800 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:40.855 INFO:tasks.ceph:Setting up mon nodes... 2024-08-16T02:18:40.855 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-16T02:18:40.855 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-16T02:18:40.953 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-16T02:18:40.956 INFO:tasks.ceph:Setting up mds nodes... 2024-08-16T02:18:40.956 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-16T02:18:40.957 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-16T02:18:41.005 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-16T02:18:41.018 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-16T02:18:41.018 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-16T02:18:41.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-16T02:18:41.042 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-16T02:18:41.042 INFO:tasks.ceph:role: osd.0 2024-08-16T02:18:41.042 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:18:41.042 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-16T02:18:41.115 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:18:41.115 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:18:41.115 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:18:41.115 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:18:41.115 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:18:41.115 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:18:41.115 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:18:41.115 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:18:41.116 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:18:41.116 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:18:41.618 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-08-16T02:18:41.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-08-16T02:18:41.621 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-16T02:18:41.731 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-16T02:18:41.762 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-16T02:18:41.828 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-16T02:18:41.828 INFO:tasks.ceph:role: osd.1 2024-08-16T02:18:41.828 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:18:41.828 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-16T02:18:41.899 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:18:41.899 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:18:41.899 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:18:41.900 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:18:41.900 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:18:41.900 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:18:41.900 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:18:41.900 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:18:41.900 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:18:41.900 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:18:42.352 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-08-16T02:18:42.355 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-08-16T02:18:42.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-16T02:18:42.428 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-16T02:18:42.499 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-16T02:18:42.564 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-16T02:18:42.565 INFO:tasks.ceph:role: osd.2 2024-08-16T02:18:42.565 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:18:42.565 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-16T02:18:42.636 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:18:42.636 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:18:42.636 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:18:42.637 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:18:42.637 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:18:42.637 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:18:42.637 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:18:42.637 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:18:42.637 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:18:42.637 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:18:43.142 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-08-16T02:18:43.144 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-08-16T02:18:43.144 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-08-16T02:18:43.238 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-16T02:18:43.270 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:43.578 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:43.578 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:43,579 seastar - Reactor backend: linux-aio 2024-08-16T02:18:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:==29874==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:43,716 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:18:43.717 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:43,716 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-08-16T02:18:43.718 INFO:teuthology.orchestra.run.smithi087.stderr:DEBUG 2024-08-16 02:18:43,717 [shard 0:main] seastar - reactor::drain 2024-08-16T02:18:44.262 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:44.262 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-08-16T02:18:44.262 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-08-16T02:18:44.262 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:44.263 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:44.263 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-08-16T02:18:44.321 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:44.324 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:44,324 seastar - Reactor backend: linux-aio 2024-08-16T02:18:44.332 INFO:teuthology.orchestra.run.smithi087.stderr:==29872==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:44.594 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bd1 0x4a0bd 0x35b7c 0x3629e 0x5c2b16e 0xbac784a 0xbac7a04 0xbadf9d1 0xbb0d80d 0xbb0de54 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:44.594 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:44.790 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362f6 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:44.790 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:44.848 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8942 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:44.848 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362e5 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:44.883 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:44.941 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xe49b2 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:44.941 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:44.956 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7d1a 0x9e4b359 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:44.956 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:45.048 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113c19 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:45.049 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:45.312 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xb930cbc 0xb9311e1 0xb9315ec 0xb9318ab 0xbb027c5 0xbb0601c 0xbab9ea9 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:45.312 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:45.420 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d88fa 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:45.420 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:45.477 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d89b6 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:45.478 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:46.395 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:46.395 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-08-16T02:18:46.395 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-08-16T02:18:46.395 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:46.395 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:46.396 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-08-16T02:18:46.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-16T02:18:46.444 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:46.746 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:46.746 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:46,747 seastar - Reactor backend: linux-aio 2024-08-16T02:18:46.751 INFO:teuthology.orchestra.run.smithi087.stderr:==29919==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:46.861 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:46,861 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:18:46.861 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:46,861 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-08-16T02:18:46.861 INFO:teuthology.orchestra.run.smithi087.stderr:DEBUG 2024-08-16 02:18:46,861 [shard 0:main] seastar - reactor::drain 2024-08-16T02:18:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-08-16T02:18:47.401 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-08-16T02:18:47.401 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:47.401 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-08-16T02:18:47.464 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:47.468 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:47,468 seastar - Reactor backend: linux-aio 2024-08-16T02:18:47.475 INFO:teuthology.orchestra.run.smithi087.stderr:==29917==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:47.740 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bc7 0x4a0bd 0x35b7c 0x3629e 0xbac61ab 0xbac75e2 0xbac785c 0xbac7a04 0xbac97bf 0xbac99bb 0xbb0dc39 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:47.740 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36329 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:48.013 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7ce6 0x9e4b359 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:48.013 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:48.036 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x9e4b354 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:48.037 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:48.094 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x8b4ff 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:48.094 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:48.121 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d79e6 0x54d89ce 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:48.121 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:48.202 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36322 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:48.202 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x359a3de 0xb931838 0xbb0cd4c 0xbb0cfe8 0xbb0d1af 0xbb0d423 0xbb0d52c 0xbb0d6a8 0xbb0de54 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:48.452 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:48.534 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8393 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:48.534 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:48.592 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x9e4b348 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:48.592 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:49.556 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:49.556 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-08-16T02:18:49.556 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-08-16T02:18:49.556 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:49.556 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:49.556 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-08-16T02:18:49.582 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-16T02:18:49.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:49.905 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:49.906 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:49,906 seastar - Reactor backend: linux-aio 2024-08-16T02:18:49.911 INFO:teuthology.orchestra.run.smithi087.stderr:==29964==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:50.020 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:50,020 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:18:50.020 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:50,020 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-08-16T02:18:50.020 INFO:teuthology.orchestra.run.smithi087.stderr:DEBUG 2024-08-16 02:18:50,021 [shard 0:main] seastar - reactor::drain 2024-08-16T02:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-08-16T02:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-08-16T02:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:50.573 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-08-16T02:18:50.630 INFO:teuthology.orchestra.run.smithi087.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:50.634 INFO:teuthology.orchestra.run.smithi087.stderr:INFO 2024-08-16 02:18:50,634 seastar - Reactor backend: linux-aio 2024-08-16T02:18:50.651 INFO:teuthology.orchestra.run.smithi087.stderr:==29962==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xe49b2 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0xbac8bbc 0xbab69e6 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x967d442 0x967d7de 0x967d963 0x967da05 0x967db1e 0xbf2c63b 0xbf2c842 0xbf2c918 0xbf2ca53 0xbf2ce47 0xbf2c20a 0xbf2c2ea 0xbf2c3fe 0xbf2c0c1 0x94c7872 0x95add6e 0x92ea331 0x92ea50b 0x92ea646 0x92ea700 0x92ea7da 0x92ead26 0x92eaffe 0x92eb0b3 0x9561672 0x95618d7 0x958fd08 0x96a8aca 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:50.838 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:50.914 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xc9c12 0xca7a6 0x359447b 0x3666701 0xbb21651 0xbb219e4 0xbaad9d1 0xbaae829 0x981cebf 0x9ae705b 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:50.915 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:51.124 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d83fe 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:51.124 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:51.182 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d82dd 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:51.182 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:51.189 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x155f8 0x4a09d 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:51.189 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:51.247 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d89b6 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:51.247 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:51.289 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x8b537 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:51.290 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:51.355 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bcd 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:51.355 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:51.585 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xbb15471 0xbb158ea 0xbab6dd6 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae7108 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:51.585 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:51.667 INFO:teuthology.orchestra.run.smithi087.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d89eb 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:51.667 INFO:teuthology.orchestra.run.smithi087.stderr:kernel callstack: 2024-08-16T02:18:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:Suppressions used: 2024-08-16T02:18:52.669 INFO:teuthology.orchestra.run.smithi087.stderr: count bytes template 2024-08-16T02:18:52.669 INFO:teuthology.orchestra.run.smithi087.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:----------------------------------------------------- 2024-08-16T02:18:52.669 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-08-16T02:18:52.694 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-16T02:18:52.719 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-16T02:18:52.719 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:52.719 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-16T02:18:52.783 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:52.783 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-16T02:18:52.851 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:52.851 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-16T02:18:52.917 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:52.917 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-16T02:18:52.981 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:52.981 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-16T02:18:53.037 INFO:tasks.ceph:Adding keys to all mons... 2024-08-16T02:18:53.037 DEBUG:teuthology.orchestra.run.smithi087:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-16T02:18:53.100 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.x] 2024-08-16T02:18:53.100 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAAt75mXujaOBAAMhd/7HOL04g+r2fQGUXl2w== 2024-08-16T02:18:53.100 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2024-08-16T02:18:53.100 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAEt75mRh+ZGxAACHPYEUhR+rKW7mK0UmTwZQ== 2024-08-16T02:18:53.100 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2024-08-16T02:18:53.100 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAHt75msBZMJBAAdZ3/R/gKuE2NHakBRSrvbQ== 2024-08-16T02:18:53.100 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2024-08-16T02:18:53.100 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAKt75mNQB9LhAAjyuNZvxCE03zxAbBERcgHw== 2024-08-16T02:18:53.101 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2024-08-16T02:18:53.101 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQABt75mBxlZABAALAzmGPhqy/jYAsXX0uHHyw== 2024-08-16T02:18:53.101 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-16T02:18:53.202 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:18:53.263 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:18:53.323 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:18:53.382 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-16T02:18:53.446 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-16T02:18:53.446 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-16T02:18:53.471 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-16T02:18:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:53.578+0000 7fd27d8d2c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:53.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:53.578+0000 7fd27d8d2c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:53.596 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-16T02:18:53.620 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:53.675 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-16T02:18:53.675 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-16T02:18:53.675 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-16T02:18:53.717 INFO:tasks.ceph.mon.a:Started 2024-08-16T02:18:53.717 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-16T02:18:53.718 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-16T02:18:53.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-16T02:18:53.720 INFO:tasks.ceph.mgr.x:Started 2024-08-16T02:18:53.720 DEBUG:tasks.ceph:set 0 configs 2024-08-16T02:18:53.720 INFO:tasks.ceph:Setting crush tunables to default 2024-08-16T02:18:53.720 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd crush tunables default 2024-08-16T02:18:53.801 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-08-16T02:18:53.800+0000 7ff2f4a66c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:53.801 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-08-16T02:18:53.800+0000 7ff2f4a66c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:53.801 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-08-16T02:18:53.800+0000 7ff2f4a66c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:53.801 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-08-16T02:18:53.800+0000 7f60bd773000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:53.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:53.824+0000 7f625ac17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:53.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:53.835+0000 7f625ac17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:53.908 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2024-08-16T02:18:53.919 INFO:tasks.ceph:check_enable_crimson: True 2024-08-16T02:18:53.920 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-08-16T02:18:53.920 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-08-16T02:18:54.007 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-08-16T02:18:54.005+0000 7f60bd773000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:54.007 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-08-16T02:18:54.006+0000 7f60bd773000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:54.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:54.020+0000 7f9a4146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:54.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:54.029+0000 7f9a4146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:54.115 INFO:tasks.ceph.mon.a.smithi087.stderr:2024-08-16T02:18:54.114+0000 7ff2eb3f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-08-16T02:18:54.115 INFO:tasks.ceph.mon.a.smithi087.stderr:Please be aware that this feature is experimental, untested, 2024-08-16T02:18:54.115 INFO:tasks.ceph.mon.a.smithi087.stderr:unsupported, and may result in data corruption, data loss, 2024-08-16T02:18:54.115 INFO:tasks.ceph.mon.a.smithi087.stderr:and/or irreparable damage to your cluster. Do not use 2024-08-16T02:18:54.115 INFO:tasks.ceph.mon.a.smithi087.stderr:feature with important data. 2024-08-16T02:18:54.115 INFO:tasks.ceph.mon.a.smithi087.stderr: 2024-08-16T02:18:54.920 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-16T02:18:54.921 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:54.921 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-16T02:18:54.944 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:54.945 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-16T02:18:55.010 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:18:55.011 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-16T02:18:55.076 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 8de1315e-0175-4dc6-b305-fb3198ae930f 0 2024-08-16T02:18:55.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:55.218+0000 7f338d192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:55.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:55.227+0000 7f338d192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:55.311 INFO:teuthology.orchestra.run.smithi087.stdout:0 2024-08-16T02:18:55.323 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 169faddf-7bed-4cf1-9d94-92689aff3df9 1 2024-08-16T02:18:55.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:55.424+0000 7f5b95017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:55.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:55.433+0000 7f5b95017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:55.516 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-08-16T02:18:55.528 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new e18ba384-d888-4ff8-89e1-3b1799a1a55d 2 2024-08-16T02:18:55.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:55.629+0000 7fab4d8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:55.639+0000 7fab4d8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:55.723 INFO:teuthology.orchestra.run.smithi087.stdout:2 2024-08-16T02:18:55.735 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-08-16T02:18:55.764 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-16T02:18:55.764 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-08-16T02:18:55.806 INFO:tasks.ceph.osd.0:Started 2024-08-16T02:18:55.806 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-08-16T02:18:55.835 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-16T02:18:55.835 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-08-16T02:18:55.877 INFO:tasks.ceph.osd.1:Started 2024-08-16T02:18:55.877 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-08-16T02:18:55.909 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-16T02:18:55.909 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-08-16T02:18:55.951 INFO:tasks.ceph.osd.2:Started 2024-08-16T02:18:55.952 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:18:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:56.085+0000 7fe740c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:56.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:18:56.093+0000 7fe740c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:56.168 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:18:56.168 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":6,"fsid":"a2704e77-df6d-4e59-a0c8-3375fbc7352b","created":"2024-08-16T02:18:53.821789+0000","modified":"2024-08-16T02:18:55.722536+0000","last_up_change":"0.000000","last_in_change":"2024-08-16T02:18:55.722536+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"8de1315e-0175-4dc6-b305-fb3198ae930f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"169faddf-7bed-4cf1-9d94-92689aff3df9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e18ba384-d888-4ff8-89e1-3b1799a1a55d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:18:56.187 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-16T02:18:56.187 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-16T02:18:56.489 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-16T02:18:56.593 INFO:teuthology.misc.health.smithi087.stderr:2024-08-16T02:18:56.588+0000 7fd6ee02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:56.597 INFO:teuthology.misc.health.smithi087.stderr:2024-08-16T02:18:56.596+0000 7fd6ee02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:56.670 INFO:teuthology.misc.health.smithi087.stdout: 2024-08-16T02:18:56.670 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":6,"fsid":"a2704e77-df6d-4e59-a0c8-3375fbc7352b","created":"2024-08-16T02:18:53.821789+0000","modified":"2024-08-16T02:18:55.722536+0000","last_up_change":"0.000000","last_in_change":"2024-08-16T02:18:55.722536+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"8de1315e-0175-4dc6-b305-fb3198ae930f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"169faddf-7bed-4cf1-9d94-92689aff3df9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e18ba384-d888-4ff8-89e1-3b1799a1a55d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:18:56.686 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-16T02:18:59.615 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-08-16T02:18:59.614+0000 7f605b598640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-16T02:19:02.988 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-16T02:19:03.096 INFO:teuthology.misc.health.smithi087.stderr:2024-08-16T02:19:03.091+0000 7f5d2de20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:03.099 INFO:teuthology.misc.health.smithi087.stderr:2024-08-16T02:19:03.099+0000 7f5d2de20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:03.300 INFO:teuthology.misc.health.smithi087.stdout: 2024-08-16T02:19:03.300 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":10,"fsid":"a2704e77-df6d-4e59-a0c8-3375fbc7352b","created":"2024-08-16T02:18:53.821789+0000","modified":"2024-08-16T02:19:02.675182+0000","last_up_change":"2024-08-16T02:19:01.666431+0000","last_in_change":"2024-08-16T02:18:55.722536+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:19:02.630075+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8de1315e-0175-4dc6-b305-fb3198ae930f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":102364345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":102364345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":102364345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":102364345}]},"public_addr":"172.21.15.87:6805/102364345","cluster_addr":"172.21.15.87:6802/102364345","heartbeat_back_addr":"172.21.15.87:6811/102364345","heartbeat_front_addr":"172.21.15.87:6810/102364345","state":["exists","up"]},{"osd":1,"uuid":"169faddf-7bed-4cf1-9d94-92689aff3df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1247754176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1247754176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":1247754176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1247754176}]},"public_addr":"172.21.15.87:6804/1247754176","cluster_addr":"172.21.15.87:6803/1247754176","heartbeat_back_addr":"172.21.15.87:6807/1247754176","heartbeat_front_addr":"172.21.15.87:6808/1247754176","state":["exists","up"]},{"osd":2,"uuid":"e18ba384-d888-4ff8-89e1-3b1799a1a55d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":1342115893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":1342115893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":1342115893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1342115893}]},"public_addr":"172.21.15.87:6809/1342115893","cluster_addr":"172.21.15.87:6812/1342115893","heartbeat_back_addr":"172.21.15.87:6813/1342115893","heartbeat_front_addr":"172.21.15.87:6806/1342115893","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:19:03.311 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-08-16T02:19:03.311 INFO:tasks.ceph:Creating RBD pool 2024-08-16T02:19:03.311 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-16T02:19:03.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:03.414+0000 7fe910a57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:03.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:03.425+0000 7fe910a57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:03.682 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'rbd' created 2024-08-16T02:19:03.703 DEBUG:teuthology.orchestra.run.smithi087:> rbd --cluster ceph pool init rbd 2024-08-16T02:19:03.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:03.803+0000 7f2f162c98c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:03.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:03.803+0000 7f2f162c98c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:03.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:03.810+0000 7f2f162c98c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:06.750 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-16T02:19:06.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-08-16T02:19:06.751 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-16T02:19:06.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:06.854+0000 7fc0fd792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:06.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:06.863+0000 7fc0fd792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:07.057 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:19:07.070 INFO:teuthology.orchestra.run.smithi087.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-08-16T02:19:07.070 INFO:tasks.ceph_manager:config epoch is 1 2024-08-16T02:19:07.070 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-16T02:19:07.070 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-16T02:19:07.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-16T02:19:07.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:07.173+0000 7ff721c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:07.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:07.180+0000 7ff721c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:07.428 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:19:07.440 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1329919188},{"type":"v1","addr":"172.21.15.87:6801","nonce":1329919188}]},"active_addr":"172.21.15.87:6801/1329919188","active_change":"2024-08-16T02:18:58.599221+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":2353177912}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":3970002019}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":1579876399}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":3306277900}]}]} 2024-08-16T02:19:07.444 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-16T02:19:07.444 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-16T02:19:07.444 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:19:07.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:07.547+0000 7fbe8cf49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:07.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:07.555+0000 7fbe8cf49640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:07.753 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:19:07.753 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":14,"fsid":"a2704e77-df6d-4e59-a0c8-3375fbc7352b","created":"2024-08-16T02:18:53.821789+0000","modified":"2024-08-16T02:19:06.695403+0000","last_up_change":"2024-08-16T02:19:01.666431+0000","last_in_change":"2024-08-16T02:18:55.722536+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:19:02.630075+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:19:03.622648+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8de1315e-0175-4dc6-b305-fb3198ae930f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":102364345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":102364345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":102364345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":102364345}]},"public_addr":"172.21.15.87:6805/102364345","cluster_addr":"172.21.15.87:6802/102364345","heartbeat_back_addr":"172.21.15.87:6811/102364345","heartbeat_front_addr":"172.21.15.87:6810/102364345","state":["exists","up"]},{"osd":1,"uuid":"169faddf-7bed-4cf1-9d94-92689aff3df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1247754176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1247754176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":1247754176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1247754176}]},"public_addr":"172.21.15.87:6804/1247754176","cluster_addr":"172.21.15.87:6803/1247754176","heartbeat_back_addr":"172.21.15.87:6807/1247754176","heartbeat_front_addr":"172.21.15.87:6808/1247754176","state":["exists","up"]},{"osd":2,"uuid":"e18ba384-d888-4ff8-89e1-3b1799a1a55d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":1342115893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":1342115893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":1342115893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1342115893}]},"public_addr":"172.21.15.87:6809/1342115893","cluster_addr":"172.21.15.87:6812/1342115893","heartbeat_back_addr":"172.21.15.87:6813/1342115893","heartbeat_front_addr":"172.21.15.87:6806/1342115893","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:19:07.767 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-16T02:19:07.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:19:07.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:07.870+0000 7fa7b4457640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:07.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:07.877+0000 7fa7b4457640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:19:08.073 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":14,"fsid":"a2704e77-df6d-4e59-a0c8-3375fbc7352b","created":"2024-08-16T02:18:53.821789+0000","modified":"2024-08-16T02:19:06.695403+0000","last_up_change":"2024-08-16T02:19:01.666431+0000","last_in_change":"2024-08-16T02:18:55.722536+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:19:02.630075+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:19:03.622648+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8de1315e-0175-4dc6-b305-fb3198ae930f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":102364345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":102364345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":102364345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":102364345}]},"public_addr":"172.21.15.87:6805/102364345","cluster_addr":"172.21.15.87:6802/102364345","heartbeat_back_addr":"172.21.15.87:6811/102364345","heartbeat_front_addr":"172.21.15.87:6810/102364345","state":["exists","up"]},{"osd":1,"uuid":"169faddf-7bed-4cf1-9d94-92689aff3df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1247754176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1247754176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":1247754176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1247754176}]},"public_addr":"172.21.15.87:6804/1247754176","cluster_addr":"172.21.15.87:6803/1247754176","heartbeat_back_addr":"172.21.15.87:6807/1247754176","heartbeat_front_addr":"172.21.15.87:6808/1247754176","state":["exists","up"]},{"osd":2,"uuid":"e18ba384-d888-4ff8-89e1-3b1799a1a55d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":1342115893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":1342115893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":1342115893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1342115893}]},"public_addr":"172.21.15.87:6809/1342115893","cluster_addr":"172.21.15.87:6812/1342115893","heartbeat_back_addr":"172.21.15.87:6813/1342115893","heartbeat_front_addr":"172.21.15.87:6806/1342115893","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:19:08.086 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-16T02:19:08.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-16T02:19:08.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-16T02:19:08.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.195+0000 7f1c5ecad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.195+0000 7fbcfeead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.196+0000 7f8619264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.204+0000 7fbcfeead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.205+0000 7f8619264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.205+0000 7f1c5ecad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.239 INFO:teuthology.orchestra.run.smithi087.stdout:34359738375 2024-08-16T02:19:08.239 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-16T02:19:08.240 INFO:teuthology.orchestra.run.smithi087.stdout:38654705670 2024-08-16T02:19:08.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-16T02:19:08.242 INFO:teuthology.orchestra.run.smithi087.stdout:34359738375 2024-08-16T02:19:08.242 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-16T02:19:08.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.346+0000 7f2fd9752640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.356+0000 7f2fd9752640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.391+0000 7f4d8caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.391+0000 7eff7f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.399+0000 7eff7f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.400+0000 7f4d8caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.603 INFO:teuthology.orchestra.run.smithi087.stdout:34359738375 2024-08-16T02:19:08.604 INFO:teuthology.orchestra.run.smithi087.stdout:34359738375 2024-08-16T02:19:08.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.0 2024-08-16T02:19:08.617 DEBUG:teuthology.parallel:result is None 2024-08-16T02:19:08.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2024-08-16T02:19:08.619 DEBUG:teuthology.parallel:result is None 2024-08-16T02:19:08.641 INFO:teuthology.orchestra.run.smithi087.stdout:38654705670 2024-08-16T02:19:08.655 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2024-08-16T02:19:08.655 DEBUG:teuthology.parallel:result is None 2024-08-16T02:19:08.655 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-16T02:19:08.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:19:08.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.799+0000 7f896ca17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:08.806+0000 7f896ca17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:08.999 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:19:08.999 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-08-16T02:19:09.011 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-08-16T02:19:08.605647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3348,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889836,"statfs":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.371727"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.719093+0000","last_change":"2024-08-16T02:19:06.726892+0000","last_active":"2024-08-16T02:19:06.719093+0000","last_peered":"2024-08-16T02:19:06.719093+0000","last_clean":"2024-08-16T02:19:06.719093+0000","last_became_active":"2024-08-16T02:19:04.800574+0000","last_became_peered":"2024-08-16T02:19:04.800574+0000","last_unstale":"2024-08-16T02:19:06.719093+0000","last_undegraded":"2024-08-16T02:19:06.719093+0000","last_fullsized":"2024-08-16T02:19:06.719093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.724084+0000","last_change":"2024-08-16T02:19:06.731654+0000","last_active":"2024-08-16T02:19:06.724084+0000","last_peered":"2024-08-16T02:19:06.724084+0000","last_clean":"2024-08-16T02:19:06.724084+0000","last_became_active":"2024-08-16T02:19:04.856804+0000","last_became_peered":"2024-08-16T02:19:04.856804+0000","last_unstale":"2024-08-16T02:19:06.724084+0000","last_undegraded":"2024-08-16T02:19:06.724084+0000","last_fullsized":"2024-08-16T02:19:06.724084+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.719785+0000","last_change":"2024-08-16T02:19:06.727198+0000","last_active":"2024-08-16T02:19:06.719785+0000","last_peered":"2024-08-16T02:19:06.719785+0000","last_clean":"2024-08-16T02:19:06.719785+0000","last_became_active":"2024-08-16T02:19:04.811355+0000","last_became_peered":"2024-08-16T02:19:04.811355+0000","last_unstale":"2024-08-16T02:19:06.719785+0000","last_undegraded":"2024-08-16T02:19:06.719785+0000","last_fullsized":"2024-08-16T02:19:06.719785+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.718416+0000","last_change":"2024-08-16T02:19:06.727430+0000","last_active":"2024-08-16T02:19:06.718416+0000","last_peered":"2024-08-16T02:19:06.718416+0000","last_clean":"2024-08-16T02:19:06.718416+0000","last_became_active":"2024-08-16T02:19:04.829841+0000","last_became_peered":"2024-08-16T02:19:04.829841+0000","last_unstale":"2024-08-16T02:19:06.718416+0000","last_undegraded":"2024-08-16T02:19:06.718416+0000","last_fullsized":"2024-08-16T02:19:06.718416+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.726589+0000","last_change":"2024-08-16T02:19:06.726589+0000","last_active":"2024-08-16T02:19:06.726589+0000","last_peered":"2024-08-16T02:19:06.726589+0000","last_clean":"2024-08-16T02:19:06.726589+0000","last_became_active":"2024-08-16T02:19:04.859597+0000","last_became_peered":"2024-08-16T02:19:04.859597+0000","last_unstale":"2024-08-16T02:19:06.726589+0000","last_undegraded":"2024-08-16T02:19:06.726589+0000","last_fullsized":"2024-08-16T02:19:06.726589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.727741+0000","last_change":"2024-08-16T02:19:06.731444+0000","last_active":"2024-08-16T02:19:06.727741+0000","last_peered":"2024-08-16T02:19:06.727741+0000","last_clean":"2024-08-16T02:19:06.727741+0000","last_became_active":"2024-08-16T02:19:04.864594+0000","last_became_peered":"2024-08-16T02:19:04.864594+0000","last_unstale":"2024-08-16T02:19:06.727741+0000","last_undegraded":"2024-08-16T02:19:06.727741+0000","last_fullsized":"2024-08-16T02:19:06.727741+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.723169+0000","last_change":"2024-08-16T02:19:06.726645+0000","last_active":"2024-08-16T02:19:06.723169+0000","last_peered":"2024-08-16T02:19:06.723169+0000","last_clean":"2024-08-16T02:19:06.723169+0000","last_became_active":"2024-08-16T02:19:04.825684+0000","last_became_peered":"2024-08-16T02:19:04.825684+0000","last_unstale":"2024-08-16T02:19:06.723169+0000","last_undegraded":"2024-08-16T02:19:06.723169+0000","last_fullsized":"2024-08-16T02:19:06.723169+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.725247+0000","last_change":"2024-08-16T02:19:06.731349+0000","last_active":"2024-08-16T02:19:06.725247+0000","last_peered":"2024-08-16T02:19:06.725247+0000","last_clean":"2024-08-16T02:19:06.725247+0000","last_became_active":"2024-08-16T02:19:04.824906+0000","last_became_peered":"2024-08-16T02:19:04.824906+0000","last_unstale":"2024-08-16T02:19:06.725247+0000","last_undegraded":"2024-08-16T02:19:06.725247+0000","last_fullsized":"2024-08-16T02:19:06.725247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.715775+0000","last_change":"2024-08-16T02:19:03.779526+0000","last_active":"2024-08-16T02:19:06.715775+0000","last_peered":"2024-08-16T02:19:06.715775+0000","last_clean":"2024-08-16T02:19:06.715775+0000","last_became_active":"2024-08-16T02:19:03.745241+0000","last_became_peered":"2024-08-16T02:19:03.745241+0000","last_unstale":"2024-08-16T02:19:06.715775+0000","last_undegraded":"2024-08-16T02:19:06.715775+0000","last_fullsized":"2024-08-16T02:19:06.715775+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_clean_scrub_stamp":"2024-08-16T02:19:02.675182+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":428,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297300,"statfs":{"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1588,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296140,"statfs":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296396,"statfs":{"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:19:09.012 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:19:09.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:09.116+0000 7f31e1b52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:09.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:09.123+0000 7f31e1b52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:09.320 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:19:09.320 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-08-16T02:19:09.333 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-08-16T02:19:08.605647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3348,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889836,"statfs":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.371727"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.719093+0000","last_change":"2024-08-16T02:19:06.726892+0000","last_active":"2024-08-16T02:19:06.719093+0000","last_peered":"2024-08-16T02:19:06.719093+0000","last_clean":"2024-08-16T02:19:06.719093+0000","last_became_active":"2024-08-16T02:19:04.800574+0000","last_became_peered":"2024-08-16T02:19:04.800574+0000","last_unstale":"2024-08-16T02:19:06.719093+0000","last_undegraded":"2024-08-16T02:19:06.719093+0000","last_fullsized":"2024-08-16T02:19:06.719093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.724084+0000","last_change":"2024-08-16T02:19:06.731654+0000","last_active":"2024-08-16T02:19:06.724084+0000","last_peered":"2024-08-16T02:19:06.724084+0000","last_clean":"2024-08-16T02:19:06.724084+0000","last_became_active":"2024-08-16T02:19:04.856804+0000","last_became_peered":"2024-08-16T02:19:04.856804+0000","last_unstale":"2024-08-16T02:19:06.724084+0000","last_undegraded":"2024-08-16T02:19:06.724084+0000","last_fullsized":"2024-08-16T02:19:06.724084+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.719785+0000","last_change":"2024-08-16T02:19:06.727198+0000","last_active":"2024-08-16T02:19:06.719785+0000","last_peered":"2024-08-16T02:19:06.719785+0000","last_clean":"2024-08-16T02:19:06.719785+0000","last_became_active":"2024-08-16T02:19:04.811355+0000","last_became_peered":"2024-08-16T02:19:04.811355+0000","last_unstale":"2024-08-16T02:19:06.719785+0000","last_undegraded":"2024-08-16T02:19:06.719785+0000","last_fullsized":"2024-08-16T02:19:06.719785+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.718416+0000","last_change":"2024-08-16T02:19:06.727430+0000","last_active":"2024-08-16T02:19:06.718416+0000","last_peered":"2024-08-16T02:19:06.718416+0000","last_clean":"2024-08-16T02:19:06.718416+0000","last_became_active":"2024-08-16T02:19:04.829841+0000","last_became_peered":"2024-08-16T02:19:04.829841+0000","last_unstale":"2024-08-16T02:19:06.718416+0000","last_undegraded":"2024-08-16T02:19:06.718416+0000","last_fullsized":"2024-08-16T02:19:06.718416+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.726589+0000","last_change":"2024-08-16T02:19:06.726589+0000","last_active":"2024-08-16T02:19:06.726589+0000","last_peered":"2024-08-16T02:19:06.726589+0000","last_clean":"2024-08-16T02:19:06.726589+0000","last_became_active":"2024-08-16T02:19:04.859597+0000","last_became_peered":"2024-08-16T02:19:04.859597+0000","last_unstale":"2024-08-16T02:19:06.726589+0000","last_undegraded":"2024-08-16T02:19:06.726589+0000","last_fullsized":"2024-08-16T02:19:06.726589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.727741+0000","last_change":"2024-08-16T02:19:06.731444+0000","last_active":"2024-08-16T02:19:06.727741+0000","last_peered":"2024-08-16T02:19:06.727741+0000","last_clean":"2024-08-16T02:19:06.727741+0000","last_became_active":"2024-08-16T02:19:04.864594+0000","last_became_peered":"2024-08-16T02:19:04.864594+0000","last_unstale":"2024-08-16T02:19:06.727741+0000","last_undegraded":"2024-08-16T02:19:06.727741+0000","last_fullsized":"2024-08-16T02:19:06.727741+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.723169+0000","last_change":"2024-08-16T02:19:06.726645+0000","last_active":"2024-08-16T02:19:06.723169+0000","last_peered":"2024-08-16T02:19:06.723169+0000","last_clean":"2024-08-16T02:19:06.723169+0000","last_became_active":"2024-08-16T02:19:04.825684+0000","last_became_peered":"2024-08-16T02:19:04.825684+0000","last_unstale":"2024-08-16T02:19:06.723169+0000","last_undegraded":"2024-08-16T02:19:06.723169+0000","last_fullsized":"2024-08-16T02:19:06.723169+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.725247+0000","last_change":"2024-08-16T02:19:06.731349+0000","last_active":"2024-08-16T02:19:06.725247+0000","last_peered":"2024-08-16T02:19:06.725247+0000","last_clean":"2024-08-16T02:19:06.725247+0000","last_became_active":"2024-08-16T02:19:04.824906+0000","last_became_peered":"2024-08-16T02:19:04.824906+0000","last_unstale":"2024-08-16T02:19:06.725247+0000","last_undegraded":"2024-08-16T02:19:06.725247+0000","last_fullsized":"2024-08-16T02:19:06.725247+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:06.715775+0000","last_change":"2024-08-16T02:19:03.779526+0000","last_active":"2024-08-16T02:19:06.715775+0000","last_peered":"2024-08-16T02:19:06.715775+0000","last_clean":"2024-08-16T02:19:06.715775+0000","last_became_active":"2024-08-16T02:19:03.745241+0000","last_became_peered":"2024-08-16T02:19:03.745241+0000","last_unstale":"2024-08-16T02:19:06.715775+0000","last_undegraded":"2024-08-16T02:19:06.715775+0000","last_fullsized":"2024-08-16T02:19:06.715775+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_clean_scrub_stamp":"2024-08-16T02:19:02.675182+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":428,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297300,"statfs":{"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1588,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296140,"statfs":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296396,"statfs":{"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:19:09.333 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-16T02:19:09.333 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-16T02:19:09.333 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-16T02:19:09.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-16T02:19:09.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:09.437+0000 7f9f18152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:19:09.443+0000 7f9f18152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:09.675 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:19:09.675 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-16T02:19:09.688 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-16T02:19:09.688 INFO:teuthology.run_tasks:Running task workunit... 2024-08-16T02:19:09.706 INFO:tasks.workunit:Pulling workunits from ref 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:19:09.706 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-16T02:19:09.707 DEBUG:teuthology.orchestra.run.smithi087:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-16T02:19:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:19:09.722 INFO:teuthology.orchestra.run.smithi087.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-16T02:19:09.722 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-16T02:19:09.778 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-16T02:19:09.779 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-16T02:19:09.834 INFO:tasks.workunit:timeout=3h 2024-08-16T02:19:09.834 INFO:tasks.workunit:cleanup=True 2024-08-16T02:19:09.834 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:19:09.891 INFO:tasks.workunit.client.0.smithi087.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-16T02:20:08.377 INFO:tasks.workunit.client.0.smithi087.stderr:Note: switching to '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'. 2024-08-16T02:20:08.378 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-08-16T02:20:08.378 INFO:tasks.workunit.client.0.smithi087.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-16T02:20:08.378 INFO:tasks.workunit.client.0.smithi087.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-16T02:20:08.378 INFO:tasks.workunit.client.0.smithi087.stderr:state without impacting any branches by switching back to a branch. 2024-08-16T02:20:08.378 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-08-16T02:20:08.378 INFO:tasks.workunit.client.0.smithi087.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-16T02:20:08.378 INFO:tasks.workunit.client.0.smithi087.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-16T02:20:08.378 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr: git switch -c 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr:Or undo this operation with: 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr: git switch - 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-08-16T02:20:08.379 INFO:tasks.workunit.client.0.smithi087.stderr:HEAD is now at 6f386a3f0b1 Merge branch 'wip-seastore-more-reports' of https://github.com/cyx1231st/ceph into wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:20:08.381 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-16T02:20:08.471 INFO:tasks.workunit.client.0.smithi087.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-16T02:20:08.473 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-16T02:20:08.473 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-16T02:20:09.069 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-16T02:20:09.124 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-16T02:20:09.162 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-16T02:20:09.164 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-16T02:20:09.164 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-16T02:20:09.202 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-16T02:20:09.205 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-08-16T02:20:09.205 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-16T02:20:09.221 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2024-08-16T02:20:09.222 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2024-08-16T02:20:09.222 DEBUG:teuthology.orchestra.run.smithi087:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-08-16T02:20:09.325 INFO:tasks.workunit.client.0.smithi087.stdout:seed 31842 2024-08-16T02:20:09.325 INFO:tasks.workunit.client.0.smithi087.stdout:Note: Google Test filter = TestLibRBD.* 2024-08-16T02:20:09.325 INFO:tasks.workunit.client.0.smithi087.stdout:[==========] Running 171 tests from 1 test suite. 2024-08-16T02:20:09.325 INFO:tasks.workunit.client.0.smithi087.stdout:[----------] Global test environment set-up. 2024-08-16T02:20:09.325 INFO:tasks.workunit.client.0.smithi087.stdout:[----------] 171 tests from TestLibRBD 2024-08-16T02:20:09.370 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-08-16T02:20:10.856 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:13.000 INFO:tasks.workunit.client.0.smithi087.stdout:image has size 2097152 and order 22 2024-08-16T02:20:13.000 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.CreateAndStat (3629 ms) 2024-08-16T02:20:13.000 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-08-16T02:20:13.000 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:780: Skipped 2024-08-16T02:20:13.000 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:13.000 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool (0 ms) 2024-08-16T02:20:13.000 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-08-16T02:20:13.001 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:13.110 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.CreateAndStatPP (110 ms) 2024-08-16T02:20:13.110 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.GetId 2024-08-16T02:20:13.110 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:13.203 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.GetId (93 ms) 2024-08-16T02:20:13.203 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-08-16T02:20:13.203 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:13.315 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.GetIdPP (112 ms) 2024-08-16T02:20:13.315 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-08-16T02:20:13.315 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:13.410 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (95 ms) 2024-08-16T02:20:13.410 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-08-16T02:20:13.410 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:13.509 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (99 ms) 2024-08-16T02:20:13.509 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-08-16T02:20:13.509 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:939: Skipped 2024-08-16T02:20:13.510 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:13.510 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp (0 ms) 2024-08-16T02:20:13.510 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-08-16T02:20:13.510 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:962: Skipped 2024-08-16T02:20:13.510 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:13.510 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP (0 ms) 2024-08-16T02:20:13.510 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.OpenAio 2024-08-16T02:20:13.510 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:13.604 INFO:tasks.workunit.client.0.smithi087.stdout:image has size 2097152 and order 22 2024-08-16T02:20:13.604 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.OpenAio (95 ms) 2024-08-16T02:20:13.604 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-08-16T02:20:13.616 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.OpenAioFail (12 ms) 2024-08-16T02:20:13.616 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-08-16T02:20:13.616 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:13.779 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.OpenAioPP (163 ms) 2024-08-16T02:20:13.779 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-08-16T02:20:13.792 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.OpenAioFailPP (14 ms) 2024-08-16T02:20:13.793 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-08-16T02:20:13.793 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:14.023 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.ResizeAndStat (230 ms) 2024-08-16T02:20:14.023 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-08-16T02:20:14.023 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:14.217 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (194 ms) 2024-08-16T02:20:14.217 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-08-16T02:20:14.217 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:14.448 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (231 ms) 2024-08-16T02:20:14.448 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-08-16T02:20:14.448 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:14.693 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (245 ms) 2024-08-16T02:20:14.694 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-08-16T02:20:16.095 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:18.012 INFO:tasks.workunit.client.0.smithi087.stdout:image: image15 2024-08-16T02:20:18.012 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image15 2024-08-16T02:20:18.012 INFO:tasks.workunit.client.0.smithi087.stdout:found image15 2024-08-16T02:20:18.012 INFO:tasks.workunit.client.0.smithi087.stdout:erased image15 2024-08-16T02:20:18.012 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:18.271 INFO:tasks.workunit.client.0.smithi087.stdout:image: image15 2024-08-16T02:20:18.271 INFO:tasks.workunit.client.0.smithi087.stdout:image: image16 2024-08-16T02:20:18.271 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image15 2024-08-16T02:20:18.271 INFO:tasks.workunit.client.0.smithi087.stdout:found image15 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:erased image15 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image16 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:found image16 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:erased image16 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:image: image16 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image16 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:found image16 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:erased image16 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (3578 ms) 2024-08-16T02:20:18.272 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-08-16T02:20:19.903 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:22.064 INFO:tasks.workunit.client.0.smithi087.stdout:num images is: 1 2024-08-16T02:20:22.064 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:22.064 INFO:tasks.workunit.client.0.smithi087.stdout:image: image17 2024-08-16T02:20:22.064 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image17 2024-08-16T02:20:22.064 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:22.123 INFO:tasks.workunit.client.0.smithi087.stdout:num images is: 2 2024-08-16T02:20:22.123 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 2 2024-08-16T02:20:22.123 INFO:tasks.workunit.client.0.smithi087.stdout:image: image17 2024-08-16T02:20:22.124 INFO:tasks.workunit.client.0.smithi087.stdout:image: image18 2024-08-16T02:20:22.124 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image17 2024-08-16T02:20:22.124 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image18 2024-08-16T02:20:22.246 INFO:tasks.workunit.client.0.smithi087.stdout:num images is: 1 2024-08-16T02:20:22.246 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:22.247 INFO:tasks.workunit.client.0.smithi087.stdout:image: image18 2024-08-16T02:20:22.247 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image18 2024-08-16T02:20:22.247 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (3975 ms) 2024-08-16T02:20:22.247 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCopy 2024-08-16T02:20:23.926 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:31.775 INFO:tasks.workunit.client.0.smithi087.stdout:image: image19 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image19 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:found image19 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:erased image19 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:image: image19 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:image: image20 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image19 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:found image19 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:erased image19 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image20 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:found image20 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:erased image20 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:0.00% done 2024-08-16T02:20:31.776 INFO:tasks.workunit.client.0.smithi087.stdout:100.00% done 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:image: image19 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:image: image20 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:image: image21 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image19 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:found image19 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:erased image19 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image20 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:found image20 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:erased image20 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image21 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:found image21 2024-08-16T02:20:31.777 INFO:tasks.workunit.client.0.smithi087.stdout:erased image21 2024-08-16T02:20:31.778 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCopy (9529 ms) 2024-08-16T02:20:31.778 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-08-16T02:20:32.966 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:38.009 INFO:tasks.workunit.client.0.smithi087.stdout:num images is: 1 2024-08-16T02:20:38.009 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:38.009 INFO:tasks.workunit.client.0.smithi087.stdout:image: image22 2024-08-16T02:20:38.009 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image22 2024-08-16T02:20:40.475 INFO:tasks.workunit.client.0.smithi087.stdout:num images is: 2 2024-08-16T02:20:40.475 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 2 2024-08-16T02:20:40.475 INFO:tasks.workunit.client.0.smithi087.stdout:image: image22 2024-08-16T02:20:40.475 INFO:tasks.workunit.client.0.smithi087.stdout:image: image23 2024-08-16T02:20:40.475 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image22 2024-08-16T02:20:40.475 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image23 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:0.00% done 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:100.00% done 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:num images is: 3 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 3 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:image: image22 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:image: image23 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:image: image24 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image22 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image23 2024-08-16T02:20:41.291 INFO:tasks.workunit.client.0.smithi087.stdout:expected = image24 2024-08-16T02:20:41.944 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCopyPP (10169 ms) 2024-08-16T02:20:41.944 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:1578: Skipped 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy (0 ms) 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:1745: Skipped 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP (0 ms) 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-08-16T02:20:41.945 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:46.215 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 1 2024-08-16T02:20:46.215 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:46.215 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap1 2024-08-16T02:20:46.215 INFO:tasks.workunit.client.0.smithi087.stdout:found snap1 with size 2097152 2024-08-16T02:20:46.215 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 2 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 2 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap1 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap2 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:found snap1 with size 2097152 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:found snap2 with size 4194304 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 1 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap2 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:found snap2 with size 4194304 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 0 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 0 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (4271 ms) 2024-08-16T02:20:46.216 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-08-16T02:20:46.217 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:1890: Skipped 2024-08-16T02:20:46.217 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:46.217 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp (0 ms) 2024-08-16T02:20:46.217 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-08-16T02:20:46.217 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:47.193 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 1 2024-08-16T02:20:47.193 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:47.193 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap1 2024-08-16T02:20:47.193 INFO:tasks.workunit.client.0.smithi087.stdout:found snap1 with size 2097152 2024-08-16T02:20:48.217 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 2 2024-08-16T02:20:48.217 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 2 2024-08-16T02:20:48.217 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap1 2024-08-16T02:20:48.217 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap2 2024-08-16T02:20:48.217 INFO:tasks.workunit.client.0.smithi087.stdout:found snap1 with size 2097152 2024-08-16T02:20:48.217 INFO:tasks.workunit.client.0.smithi087.stdout:found snap2 with size 4194304 2024-08-16T02:20:49.230 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 1 2024-08-16T02:20:49.230 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:49.230 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap2 2024-08-16T02:20:49.230 INFO:tasks.workunit.client.0.smithi087.stdout:found snap2 with size 4194304 2024-08-16T02:20:50.218 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 0 2024-08-16T02:20:50.218 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 0 2024-08-16T02:20:50.233 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4018 ms) 2024-08-16T02:20:50.233 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-08-16T02:20:50.233 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:56.246 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 0 2024-08-16T02:20:56.247 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 0 2024-08-16T02:20:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6036 ms) 2024-08-16T02:20:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-08-16T02:20:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:20:57.237 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 1 2024-08-16T02:20:57.237 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:57.237 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap1 2024-08-16T02:20:57.237 INFO:tasks.workunit.client.0.smithi087.stdout:found snap1 with size 2097152 2024-08-16T02:20:58.213 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 2 2024-08-16T02:20:58.213 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 2 2024-08-16T02:20:58.213 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap1 2024-08-16T02:20:58.213 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap2 2024-08-16T02:20:58.213 INFO:tasks.workunit.client.0.smithi087.stdout:found snap1 with size 2097152 2024-08-16T02:20:58.214 INFO:tasks.workunit.client.0.smithi087.stdout:found snap2 with size 4194304 2024-08-16T02:20:58.257 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 2 2024-08-16T02:20:58.257 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 2 2024-08-16T02:20:58.257 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap1-rename 2024-08-16T02:20:58.257 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap2 2024-08-16T02:20:58.258 INFO:tasks.workunit.client.0.smithi087.stdout:found snap1-rename with size 2097152 2024-08-16T02:20:58.258 INFO:tasks.workunit.client.0.smithi087.stdout:found snap2 with size 4194304 2024-08-16T02:20:59.329 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 1 2024-08-16T02:20:59.330 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:20:59.330 INFO:tasks.workunit.client.0.smithi087.stdout:snap: snap2-rename 2024-08-16T02:20:59.330 INFO:tasks.workunit.client.0.smithi087.stdout:found snap2-rename with size 4194304 2024-08-16T02:21:00.271 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 0 2024-08-16T02:21:00.271 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 0 2024-08-16T02:21:00.292 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (4023 ms) 2024-08-16T02:21:00.292 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-08-16T02:21:02.085 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:21:02.086 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:21:02.086 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:21:02.086 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:21:04.967 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (4675 ms) 2024-08-16T02:21:04.967 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-08-16T02:21:09.341 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:21:29.574 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.RemoveFullTry (24607 ms) 2024-08-16T02:21:29.574 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-08-16T02:21:29.574 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2227: Skipped 2024-08-16T02:21:29.574 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:21:29.574 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool (0 ms) 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2247: Skipped 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespace (0 ms) 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2269: Skipped 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespaceDataPool (0 ms) 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestIO 2024-08-16T02:21:29.575 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:21:56.266 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:21:56.266 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:21:56.266 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.267 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.268 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:21:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:21:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:21:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:21:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:21:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:21:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:21:56.269 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.270 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:21:56.271 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.272 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:21:56.273 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:21:56.274 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.274 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:21:56.274 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:21:56.274 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:21:56.274 INFO:tasks.workunit.client.0.smithi087.stdout:discard: 512~512 = 512 2024-08-16T02:21:56.274 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.274 INFO:tasks.workunit.client.0.smithi087.stdout:aio discard: 1536~512 = 0 2024-08-16T02:21:56.274 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.275 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.276 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.277 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.278 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.279 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.280 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.281 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.282 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.283 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:21:56.284 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.924 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.925 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.926 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.927 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:18.928 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.929 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.930 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.931 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.932 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:18.933 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:22:18.934 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:22:18.934 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:22:18.934 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:22:18.934 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestIO (49350 ms) 2024-08-16T02:22:18.934 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-08-16T02:22:18.934 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:22:55.562 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:22:55.562 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:22:55.562 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:22:55.562 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:22:55.562 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:22:55.562 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.562 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:22:55.563 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:22:55.564 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.565 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:22:55.566 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:22:55.567 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.568 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:22:55.569 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:22:55.569 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:22:55.569 INFO:tasks.workunit.client.0.smithi087.stdout:discard: 512~512 = 512 2024-08-16T02:22:55.569 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.569 INFO:tasks.workunit.client.0.smithi087.stdout:aio discard: 1536~512 = 0 2024-08-16T02:22:55.569 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.569 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.570 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.571 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.572 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.573 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.574 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.575 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:55.576 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.577 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:22:55.578 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:22:55.579 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:22:55.579 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:22:55.579 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:23:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:23:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 4 2024-08-16T02:23:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:read: 4 2024-08-16T02:23:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:read: 4 2024-08-16T02:23:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (70188 ms) 2024-08-16T02:23:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-08-16T02:23:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:23:35.214 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:35.215 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:23:35.216 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: -84 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:495: Failure 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:Expected equality of these values: 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout: len 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout: Which is: 512 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout: static_cast(written) 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout: Which is: 18446744073709551532 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:630: Failure 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:Value of: passed 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout: Actual: false 2024-08-16T02:23:35.217 INFO:tasks.workunit.client.0.smithi087.stdout:Expected: true 2024-08-16T02:23:39.865 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 4 2024-08-16T02:23:39.865 INFO:tasks.workunit.client.0.smithi087.stdout:read: 4 2024-08-16T02:23:39.865 INFO:tasks.workunit.client.0.smithi087.stdout:read: 4 2024-08-16T02:23:39.865 INFO:tasks.workunit.client.0.smithi087.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 (10753 ms) 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2471: Skipped 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption (0 ms) 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2659: Skipped 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (0 ms) 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2735: Skipped 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (0 ms) 2024-08-16T02:23:39.866 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-08-16T02:23:39.867 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:23:44.341 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4476 ms) 2024-08-16T02:23:44.341 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-08-16T02:23:44.341 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:23:50.142 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (5801 ms) 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2905: Skipped 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2985: Skipped 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch (0 ms) 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-08-16T02:23:50.143 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:02.351 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.EncryptedResize (12209 ms) 2024-08-16T02:24:02.351 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-08-16T02:24:02.351 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:3202: Skipped 2024-08-16T02:24:02.351 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:24:02.351 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData (0 ms) 2024-08-16T02:24:02.351 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-08-16T02:24:02.351 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:29.110 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:24:29.111 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:24:29.112 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:24:29.113 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:24:29.114 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.115 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:24:29.116 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:discard: 512~512 = 512 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:aio discard: 1536~512 = 0 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.117 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.118 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.119 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.120 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.121 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.122 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.123 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.124 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:29.125 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.126 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:29.127 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.123 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.123 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.123 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.123 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.123 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:52.124 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.125 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.126 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.127 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.128 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.129 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.130 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:24:52.131 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (49771 ms) 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:3947: Skipped 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO (0 ms) 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-08-16T02:24:52.132 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:52.320 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (198 ms) 2024-08-16T02:24:52.320 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-08-16T02:24:52.321 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:52.487 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (167 ms) 2024-08-16T02:24:52.487 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-08-16T02:24:52.487 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:52.660 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (174 ms) 2024-08-16T02:24:52.661 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-08-16T02:24:52.661 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:52.818 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (157 ms) 2024-08-16T02:24:52.818 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-08-16T02:24:52.819 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:53.208 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (390 ms) 2024-08-16T02:24:53.208 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-08-16T02:24:53.209 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:53.603 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (395 ms) 2024-08-16T02:24:53.604 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-08-16T02:24:53.604 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:54.266 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (663 ms) 2024-08-16T02:24:54.266 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-08-16T02:24:54.266 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:55.008 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (742 ms) 2024-08-16T02:24:55.008 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-08-16T02:24:55.009 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:24:59.470 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (4458 ms) 2024-08-16T02:24:59.626 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-08-16T02:24:59.626 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:03.660 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (4194 ms) 2024-08-16T02:25:03.660 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-08-16T02:25:03.660 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:03.810 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (150 ms) 2024-08-16T02:25:03.810 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-08-16T02:25:03.810 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:03.983 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (173 ms) 2024-08-16T02:25:03.983 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-08-16T02:25:03.983 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:04.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (160 ms) 2024-08-16T02:25:04.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-08-16T02:25:04.143 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:04.351 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (208 ms) 2024-08-16T02:25:04.351 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-08-16T02:25:04.351 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:04.492 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:04.493 INFO:tasks.workunit.client.0.smithi087.stdout:aio discard: 0~1048576 = 0 2024-08-16T02:25:04.493 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:04.493 INFO:tasks.workunit.client.0.smithi087.stdout:aio discard: 0~4194304 = 0 2024-08-16T02:25:04.493 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:04.493 INFO:tasks.workunit.client.0.smithi087.stdout:aio discard: 3145728~1048576 = 0 2024-08-16T02:25:04.493 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (141 ms) 2024-08-16T02:25:04.493 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestFUA 2024-08-16T02:25:04.493 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.047 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.048 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.049 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.050 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestFUA (555 ms) 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-08-16T02:25:05.051 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:05.142 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.142 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.142 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.142 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.143 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:25:05.144 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.427 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:start compare and write 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:compare and wrote: 512 2024-08-16T02:25:05.428 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.429 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:25:05.429 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.446 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.446 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:25:05.446 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.446 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:25:05.446 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.464 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.464 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:25:05.464 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.464 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:25:05.464 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.482 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.482 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:25:05.482 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.482 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:25:05.482 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.501 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.501 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:25:05.501 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.501 INFO:tasks.workunit.client.0.smithi087.stdout:started aio compare and write 2024-08-16T02:25:05.501 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:05.546 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:finished aio compare and write 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.547 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.552 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.552 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:25:05.552 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.553 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.553 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.558 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.559 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:25:05.559 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.559 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.559 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.564 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.564 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:25:05.564 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.564 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.564 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.570 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.570 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:25:05.570 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:05.571 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:25:05.571 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:25:05.571 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:25:05.571 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:25:05.571 INFO:tasks.workunit.client.0.smithi087.stdout:discard: 512~512 2024-08-16T02:25:05.571 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:25:27.241 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.242 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.243 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.244 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.245 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:27.246 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.247 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:27.248 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:27.248 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:27.463 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:27.463 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:27.463 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.464 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.464 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:27.464 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:27.464 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:27.604 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:27.604 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:27.605 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:28.180 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:28.181 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:28.181 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:28.181 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:28.181 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:28.181 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:28.181 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:28.647 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:28.648 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:28.648 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:28.648 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:28.648 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:28.648 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:28.648 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:29.469 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:29.470 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:29.470 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:29.470 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:29.470 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:29.470 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:29.470 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:29.989 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:29.989 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:29.989 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:29.989 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:29.990 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:31.180 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:31.180 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:31.180 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:31.180 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:31.181 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:31.181 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:31.181 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:31.999 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:31.999 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:31.999 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:31.999 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:31.999 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:31.999 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:31.999 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:33.372 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:33.372 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:33.372 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:33.373 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:33.373 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:33.373 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:33.373 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:34.265 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:34.266 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:36.026 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:36.026 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:36.026 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:36.026 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:36.027 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:36.027 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:36.027 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:37.273 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:37.273 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:37.273 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:37.273 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:37.273 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:37.274 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:37.274 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:39.253 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:39.253 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:39.253 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:39.254 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:39.254 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:39.254 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:39.254 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:40.608 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:40.608 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:40.608 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:40.608 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:40.608 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:40.608 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:40.608 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:40.609 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:40.609 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:40.609 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:40.609 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:40.609 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:40.609 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:42.921 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:42.922 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:42.922 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:42.922 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:42.922 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:42.922 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:42.922 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:44.536 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:44.536 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:44.536 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:44.536 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:44.536 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:44.536 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:44.536 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:47.115 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:47.115 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:47.115 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:47.115 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:47.115 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:47.115 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:47.115 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:48.882 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:25:48.882 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:25:48.882 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestIOPP (43835 ms) 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-08-16T02:25:48.883 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:49.010 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (128 ms) 2024-08-16T02:25:49.010 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-08-16T02:25:49.010 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:49.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (133 ms) 2024-08-16T02:25:49.143 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-08-16T02:25:49.143 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (131 ms) 2024-08-16T02:25:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-08-16T02:25:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:49.404 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (130 ms) 2024-08-16T02:25:49.404 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-08-16T02:25:49.404 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:49.572 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (168 ms) 2024-08-16T02:25:49.572 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-08-16T02:25:49.572 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:49.698 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:49.730 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (158 ms) 2024-08-16T02:25:49.730 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-08-16T02:25:49.730 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:49.875 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (145 ms) 2024-08-16T02:25:49.875 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-08-16T02:25:49.876 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:49.999 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:50.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (157 ms) 2024-08-16T02:25:50.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-08-16T02:25:50.032 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:50.196 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (164 ms) 2024-08-16T02:25:50.196 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-08-16T02:25:50.196 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:50.332 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:50.365 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (169 ms) 2024-08-16T02:25:50.365 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-08-16T02:25:50.365 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:50.566 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (201 ms) 2024-08-16T02:25:50.566 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-08-16T02:25:50.566 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:50.727 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:50.763 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (197 ms) 2024-08-16T02:25:50.763 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-08-16T02:25:50.763 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:51.228 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (465 ms) 2024-08-16T02:25:51.228 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-08-16T02:25:51.228 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:51.418 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:51.624 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (396 ms) 2024-08-16T02:25:51.624 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-08-16T02:25:51.624 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:52.069 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (445 ms) 2024-08-16T02:25:52.070 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-08-16T02:25:52.070 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:52.387 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:25:52.616 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (547 ms) 2024-08-16T02:25:52.617 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-08-16T02:25:52.617 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:25:57.001 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (4385 ms) 2024-08-16T02:25:57.001 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-08-16T02:25:57.001 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:00.990 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:01.348 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (4347 ms) 2024-08-16T02:26:01.348 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-08-16T02:26:01.348 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:01.579 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:26:01.579 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:26:01.579 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:26:01.579 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:26:01.579 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:26:01.579 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.579 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:26:01.579 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:01.595 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:01.595 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:26:01.595 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.595 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:26:01.595 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:01.611 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:01.611 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:26:01.611 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.611 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:26:01.611 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:01.628 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:01.628 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:26:01.628 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.628 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:26:01.628 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:01.644 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:01.644 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:26:01.644 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.644 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:26:01.645 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:01.663 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:01.664 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:26:01.664 INFO:tasks.workunit.client.0.smithi087.stdout:read: 512 2024-08-16T02:26:01.664 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.664 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:26:01.664 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:26:01.669 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:26:01.669 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:26:01.669 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.670 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:26:01.670 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:26:01.675 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:26:01.675 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:26:01.675 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.675 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:26:01.675 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:26:01.681 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:26:01.681 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:26:01.681 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.681 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:26:01.681 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:26:01.687 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:26:01.687 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:26:01.687 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:01.687 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:26:01.687 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:26:23.294 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:26:23.294 INFO:tasks.workunit.client.0.smithi087.stdout:finished read 2024-08-16T02:26:23.294 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:26:23.294 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.294 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.294 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 0 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 16384 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.295 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 49152 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 65536 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:26:23.296 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 98304 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 114688 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.297 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 147456 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 163840 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.298 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 196608 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 212992 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.299 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:23.300 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:23.311 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:23.311 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:23.311 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.311 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.311 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:23.311 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:23.311 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:23.519 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:23.519 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:23.519 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.519 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.519 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:23.520 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:23.520 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:23.662 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:23.662 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:23.662 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:23.662 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:23.662 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:23.662 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:23.662 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.663 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:23.663 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:23.663 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:23.663 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:23.663 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:23.663 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:24.288 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:24.288 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:24.288 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:24.288 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:24.288 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:24.289 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:24.289 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:24.720 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:24.720 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:24.720 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:24.721 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:24.721 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:24.721 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:24.721 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:25.534 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:25.534 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:25.534 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:25.535 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:25.535 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:25.535 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:25.535 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:26.069 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:26.069 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:26.070 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:27.272 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:27.272 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:27.272 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:27.272 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:27.272 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:27.272 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:27.272 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:28.060 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:28.060 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:28.060 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:28.060 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:28.060 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:28.060 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:28.060 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:29.467 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:29.467 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:29.467 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:29.467 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:29.467 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:29.467 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:29.467 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:30.367 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:30.367 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:30.367 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:30.367 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:30.367 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:30.367 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:30.367 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:32.119 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:32.119 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:32.119 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:32.119 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:32.119 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:32.119 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:32.120 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:33.373 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:33.374 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:33.374 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:33.374 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:33.374 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:33.374 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:33.374 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:35.360 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:35.360 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:35.360 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:35.360 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:35.360 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:35.360 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:35.360 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:36.720 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:36.721 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:36.721 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:39.024 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:39.025 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:39.025 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:39.025 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:39.025 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:39.025 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:39.025 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:40.636 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:40.636 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:40.636 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:40.636 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:40.636 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:40.636 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:40.636 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:43.197 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:43.197 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:43.197 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:43.197 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:43.198 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:43.198 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:43.198 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:26:44.930 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:26:44.930 INFO:tasks.workunit.client.0.smithi087.stdout:finished writesame 2024-08-16T02:26:44.930 INFO:tasks.workunit.client.0.smithi087.stdout:to verify the data 2024-08-16T02:26:44.930 INFO:tasks.workunit.client.0.smithi087.stdout:verified 2024-08-16T02:26:44.930 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:44.930 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:44.931 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:44.931 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:26:44.931 INFO:tasks.workunit.client.0.smithi087.stdout:started writesame 2024-08-16T02:26:44.931 INFO:tasks.workunit.client.0.smithi087.stdout:expected fail, finished writesame 2024-08-16T02:26:44.931 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (43582 ms) 2024-08-16T02:26:44.931 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-08-16T02:26:44.931 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:47.285 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 0 2024-08-16T02:26:47.285 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 0 2024-08-16T02:26:47.285 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 1 2024-08-16T02:26:47.285 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:26:47.285 INFO:tasks.workunit.client.0.smithi087.stdout:snap: orig 2024-08-16T02:26:47.285 INFO:tasks.workunit.client.0.smithi087.stdout:found orig with size 2097152 2024-08-16T02:26:47.285 INFO:tasks.workunit.client.0.smithi087.stdout:read: 80 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:write test data! 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 80 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 2 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 2 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:snap: orig 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:snap: written 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:found orig with size 2097152 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:found written with size 2097152 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:read: 80 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:read: 80 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:read: 80 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:write to snapshot returned -30 2024-08-16T02:26:47.286 INFO:tasks.workunit.client.0.smithi087.stdout:Read-only file system 2024-08-16T02:26:47.436 INFO:tasks.workunit.client.0.smithi087.stdout:read: 80 2024-08-16T02:26:47.436 INFO:tasks.workunit.client.0.smithi087.stdout:read: 80 2024-08-16T02:26:47.436 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 80 2024-08-16T02:26:47.436 INFO:tasks.workunit.client.0.smithi087.stdout:opening testimg@orig 2024-08-16T02:26:47.436 INFO:tasks.workunit.client.0.smithi087.stdout:read: 80 2024-08-16T02:26:47.436 INFO:tasks.workunit.client.0.smithi087.stdout:write to snapshot returned -30 2024-08-16T02:26:47.436 INFO:tasks.workunit.client.0.smithi087.stdout:Read-only file system 2024-08-16T02:26:49.273 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 2 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 2 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:snap: orig 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:snap: written 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:found orig with size 2097152 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:found written with size 2097152 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 1 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 1 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:snap: orig 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:found orig with size 2097152 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:num snaps is: 0 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:expected: 0 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (4343 ms) 2024-08-16T02:26:49.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-08-16T02:26:49.275 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:51.273 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (1999 ms) 2024-08-16T02:26:51.273 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestClone 2024-08-16T02:26:51.273 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:6444: Skipped 2024-08-16T02:26:51.273 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:26:51.273 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestClone (1 ms) 2024-08-16T02:26:51.273 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestClone2 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:6608: Skipped 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestClone2 (0 ms) 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ListChildren 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:6841: Skipped 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ListChildren (0 ms) 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:6995: Skipped 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered (0 ms) 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.LockingPP 2024-08-16T02:26:51.274 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:51.723 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.LockingPP (450 ms) 2024-08-16T02:26:51.723 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.FlushAio 2024-08-16T02:26:51.724 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:55.405 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.FlushAio (3681 ms) 2024-08-16T02:26:55.405 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-08-16T02:26:55.406 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:59.081 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.FlushAioPP (3677 ms) 2024-08-16T02:26:59.081 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-08-16T02:26:59.081 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:59.211 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (130 ms) 2024-08-16T02:26:59.211 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-08-16T02:26:59.211 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:59.396 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (184 ms) 2024-08-16T02:26:59.396 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-08-16T02:26:59.396 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:59.521 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.ZeroLengthRead (126 ms) 2024-08-16T02:26:59.521 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-08-16T02:26:59.522 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:59.786 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.LargeCacheRead (265 ms) 2024-08-16T02:26:59.786 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-08-16T02:26:59.786 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9152: Skipped 2024-08-16T02:26:59.786 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:26:59.786 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestPendingAio (0 ms) 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9264: Skipped 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyup (0 ms) 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9325: Skipped 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyupMismatch (0 ms) 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.Flatten 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9336: Skipped 2024-08-16T02:26:59.787 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:26:59.788 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.Flatten (0 ms) 2024-08-16T02:26:59.788 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.Sparsify 2024-08-16T02:26:59.788 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:26:59.972 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 32768 2024-08-16T02:26:59.972 INFO:tasks.workunit.client.0.smithi087.stdout:read: 32768 2024-08-16T02:26:59.972 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.Sparsify (186 ms) 2024-08-16T02:26:59.972 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-08-16T02:26:59.972 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:00.163 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.SparsifyPP (191 ms) 2024-08-16T02:27:00.163 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-08-16T02:27:00.163 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:05.275 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.SnapshotLimit (5112 ms) 2024-08-16T02:27:05.275 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-08-16T02:27:05.275 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:10.293 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5017 ms) 2024-08-16T02:27:10.293 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-08-16T02:27:10.293 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9523: Skipped 2024-08-16T02:27:10.293 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.293 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RebuildObjectMapViaLockOwner (0 ms) 2024-08-16T02:27:10.293 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9575: Skipped 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RenameViaLockOwner (0 ms) 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9615: Skipped 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapCreateViaLockOwner (0 ms) 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9659: Skipped 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveViaLockOwner (0 ms) 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-08-16T02:27:10.294 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9699: Skipped 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.UpdateFeaturesViaLockOwner (0 ms) 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9736: Skipped 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9777: Skipped 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapRemove2 (0 ms) 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9823: Skipped 2024-08-16T02:27:10.295 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapRenameViaLockOwner (0 ms) 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9864: Skipped 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapProtectViaLockOwner (0 ms) 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9905: Skipped 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapUnprotectViaLockOwner (0 ms) 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9949: Skipped 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.296 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.FlattenViaLockOwner (0 ms) 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9997: Skipped 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ResizeViaLockOwner (0 ms) 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10033: Skipped 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SparsifyViaLockOwner (0 ms) 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10069: Skipped 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ObjectMapConsistentSnap (0 ms) 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.Metadata 2024-08-16T02:27:10.297 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10121: Skipped 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.Metadata (0 ms) 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10310: Skipped 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.MetadataPP (0 ms) 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10391: Skipped 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.UpdateFeatures (0 ms) 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2024-08-16T02:27:10.298 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-08-16T02:27:10.299 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:10.408 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.RebuildObjectMap (116 ms) 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10584: Skipped 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RebuildNewObjectMap (0 ms) 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10609: Skipped 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CheckObjectMap (0 ms) 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-08-16T02:27:10.409 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:10.587 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.BlockingAIO (179 ms) 2024-08-16T02:27:10.587 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10737: Skipped 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockTransition (0 ms) 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10793: Skipped 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10846: Skipped 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CacheMayCopyOnWrite (0 ms) 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-08-16T02:27:10.588 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:11.344 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (757 ms) 2024-08-16T02:27:11.344 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-08-16T02:27:15.997 INFO:tasks.workunit.client.0.smithi087.stdout:0.00% done 2024-08-16T02:27:15.997 INFO:tasks.workunit.client.0.smithi087.stdout:100.00% done 2024-08-16T02:27:15.997 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestImageOptions (4653 ms) 2024-08-16T02:27:15.997 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-08-16T02:27:17.769 INFO:tasks.workunit.client.0.smithi087.stdout:0.00% done 2024-08-16T02:27:17.769 INFO:tasks.workunit.client.0.smithi087.stdout:100.00% done 2024-08-16T02:27:17.769 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (1772 ms) 2024-08-16T02:27:17.769 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-08-16T02:27:17.769 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2024-08-16T02:27:17.769 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-08-16T02:27:17.769 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-08-16T02:27:17.770 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-08-16T02:27:17.770 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:18.089 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:27:18.089 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:27:18.089 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:27:18.089 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:27:18.089 INFO:tasks.workunit.client.0.smithi087.stdout:wrote: 512 2024-08-16T02:27:18.089 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.089 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:27:18.089 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:27:18.090 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:started write 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:write completion cb called! 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 0 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:finished write 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:27:18.091 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:created completion 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:started read 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:read completion cb called! 2024-08-16T02:27:18.092 INFO:tasks.workunit.client.0.smithi087.stdout:return value is: 512 2024-08-16T02:27:18.093 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.ImagePollIO (320 ms) 2024-08-16T02:27:18.093 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.Mirror 2024-08-16T02:27:18.145 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:18.193 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:18.620 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.Mirror (531 ms) 2024-08-16T02:27:18.620 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (146 ms) 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11322: Skipped 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CreateWithMirrorEnabled (0 ms) 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11364: Skipped 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (0 ms) 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-08-16T02:27:18.767 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11410: Skipped 2024-08-16T02:27:18.768 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:18.768 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLock (0 ms) 2024-08-16T02:27:18.768 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.BreakLock 2024-08-16T02:27:18.768 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11553: Skipped 2024-08-16T02:27:18.768 INFO:tasks.workunit.client.0.smithi087.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:27:18.768 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.BreakLock (0 ms) 2024-08-16T02:27:18.768 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-08-16T02:27:18.768 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:18.877 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (109 ms) 2024-08-16T02:27:18.877 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-08-16T02:27:18.877 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-08-16T02:27:18.877 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-08-16T02:27:18.877 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11676: Skipped 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurge (0 ms) 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11722: Skipped 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (0 ms) 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11753: Skipped 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestTrashPurge (0 ms) 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-08-16T02:27:18.878 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11810: Skipped 2024-08-16T02:27:18.879 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:18.879 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndRestore (0 ms) 2024-08-16T02:27:18.879 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-08-16T02:27:18.879 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:19.087 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestListWatchers (211 ms) 2024-08-16T02:27:19.087 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-08-16T02:27:19.087 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:19.923 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.TestSetSnapById (837 ms) 2024-08-16T02:27:19.924 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.Namespaces 2024-08-16T02:27:20.221 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.Namespaces (298 ms) 2024-08-16T02:27:20.221 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-08-16T02:27:20.651 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:20.665 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.NamespacesPP (444 ms) 2024-08-16T02:27:20.665 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.Migration 2024-08-16T02:27:20.665 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:20.665 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:24.943 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.Migration (4277 ms) 2024-08-16T02:27:24.943 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-08-16T02:27:24.943 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:24.943 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.MigrationPP (4214 ms) 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12177: Skipped 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestGetAccessTimestamp (0 ms) 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12202: Skipped 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.158 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestGetModifyTimestamp (0 ms) 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12225: Skipped 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ZeroOverlapFlatten (0 ms) 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12258: Skipped 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.PoolMetadata (0 ms) 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12370: Skipped 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.PoolMetadataPP (0 ms) 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.Config 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12422: Skipped 2024-08-16T02:27:29.159 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.Config (0 ms) 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12506: Skipped 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ConfigPP (0 ms) 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12583: Skipped 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.PoolStatsPP (0 ms) 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12658: Skipped 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ImageSpec (0 ms) 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-08-16T02:27:29.160 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12853: Skipped 2024-08-16T02:27:29.161 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:29.161 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveWithChildMissing (0 ms) 2024-08-16T02:27:29.161 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-08-16T02:27:29.161 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:35.314 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.QuiesceWatch (6157 ms) 2024-08-16T02:27:35.314 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-08-16T02:27:35.314 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:41.281 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (5967 ms) 2024-08-16T02:27:41.282 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-08-16T02:27:41.282 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:13115: Skipped 2024-08-16T02:27:41.282 INFO:tasks.workunit.client.0.smithi087.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:27:41.282 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError (0 ms) 2024-08-16T02:27:41.282 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-08-16T02:27:41.282 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:27:41.372 INFO:tasks.workunit.client.0.smithi087.stderr:test quiesce is not long enough to time out 2024-08-16T02:27:50.140 INFO:tasks.workunit.client.0.smithi087.stderr:test quiesce is timed out 2024-08-16T02:27:50.163 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 0 2024-08-16T02:27:51.164 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 1 2024-08-16T02:27:52.164 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 2 2024-08-16T02:27:53.164 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 3 2024-08-16T02:27:54.165 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 4 2024-08-16T02:27:55.164 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 5 2024-08-16T02:27:56.164 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 6 2024-08-16T02:27:57.164 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 7 2024-08-16T02:27:58.164 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 8 2024-08-16T02:27:59.164 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 9 2024-08-16T02:28:00.165 INFO:tasks.workunit.client.0.smithi087.stderr:waiting for timed out ... 10 2024-08-16T02:28:11.165 INFO:tasks.workunit.client.0.smithi087.stderr:test retry succeeds 2024-08-16T02:28:14.401 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (33120 ms) 2024-08-16T02:28:14.401 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-08-16T02:28:14.401 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:28:14.660 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.WriteZeroes (259 ms) 2024-08-16T02:28:14.660 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-08-16T02:28:14.660 INFO:tasks.workunit.client.0.smithi087.stdout:using old format 2024-08-16T02:28:15.044 INFO:tasks.workunit.client.0.smithi087.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (384 ms) 2024-08-16T02:28:15.045 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.ConcurrentOperations 2024-08-16T02:28:15.045 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:13474: Skipped 2024-08-16T02:28:15.045 INFO:tasks.workunit.client.0.smithi087.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:28:15.045 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations (0 ms) 2024-08-16T02:28:15.045 INFO:tasks.workunit.client.0.smithi087.stdout:[ RUN ] TestLibRBD.FormatAndCloneFormatOptions 2024-08-16T02:28:15.045 INFO:tasks.workunit.client.0.smithi087.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:13630: Skipped 2024-08-16T02:28:15.045 INFO:tasks.workunit.client.0.smithi087.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:28:15.045 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.FormatAndCloneFormatOptions (0 ms) 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout:[----------] 171 tests from TestLibRBD (485673 ms total) 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout: 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout:[----------] Global test environment tear-down 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout:[==========] 171 tests from 1 test suite ran. (492701 ms total) 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout:[ PASSED ] 103 tests. 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] 67 tests, listed below: 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp 2024-08-16T02:28:22.027 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespace 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestClone 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestClone2 2024-08-16T02:28:22.028 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ListChildren 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestPendingAio 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyup 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.Flatten 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RenameViaLockOwner 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapCreateViaLockOwner 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveViaLockOwner 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapRemove2 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapRenameViaLockOwner 2024-08-16T02:28:22.029 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapProtectViaLockOwner 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapUnprotectViaLockOwner 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.FlattenViaLockOwner 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ResizeViaLockOwner 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SparsifyViaLockOwner 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ObjectMapConsistentSnap 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.Metadata 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.MetadataPP 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.UpdateFeatures 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.RebuildNewObjectMap 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CheckObjectMap 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockTransition 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CacheMayCopyOnWrite 2024-08-16T02:28:22.030 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.CreateWithMirrorEnabled 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLock 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.BreakLock 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurge 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestTrashPurge 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndRestore 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestGetAccessTimestamp 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.TestGetModifyTimestamp 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ZeroOverlapFlatten 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.PoolMetadata 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.PoolMetadataPP 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.Config 2024-08-16T02:28:22.031 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ConfigPP 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.PoolStatsPP 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ImageSpec 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveWithChildMissing 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ SKIPPED ] TestLibRBD.FormatAndCloneFormatOptions 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ FAILED ] 1 test, listed below: 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout: 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout: 1 FAILED TEST 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout: YOU HAVE 1 DISABLED TEST 2024-08-16T02:28:22.032 INFO:tasks.workunit.client.0.smithi087.stdout: 2024-08-16T02:28:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:28:22.035 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2024-08-16T02:28:22.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-16T02:28:22.340 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-08-16T02:28:25.573 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=22f61a978a2346cc95dfc3d47864a60c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-08-16T02:28:25.576 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-16T02:28:25.587 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-16T02:28:25.587 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:25.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:25.702+0000 7f458c792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:25.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:25.710+0000 7f458c792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:25.910 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:28:25.910 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-08-16T02:28:25.923 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-08-16T02:28:24.771991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":362632,"kb_used_data":362632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156530552,"statfs":{"total":160658620416,"available":160287285248,"internally_reserved":0,"allocated":371335168,"data_stored":371335168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168070818,"num_objects":-221,"num_object_clones":0,"num_object_copies":-442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28112,"num_read_kb":-145426,"num_write":-1526,"num_write_kb":-203274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.759186"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.079643+0000","last_change":"2024-08-16T02:19:06.726892+0000","last_active":"2024-08-16T02:28:20.079643+0000","last_peered":"2024-08-16T02:28:20.079643+0000","last_clean":"2024-08-16T02:28:20.079643+0000","last_became_active":"2024-08-16T02:19:04.800574+0000","last_became_peered":"2024-08-16T02:19:04.800574+0000","last_unstale":"2024-08-16T02:28:20.079643+0000","last_undegraded":"2024-08-16T02:28:20.079643+0000","last_fullsized":"2024-08-16T02:28:20.079643+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.084523+0000","last_change":"2024-08-16T02:19:06.731654+0000","last_active":"2024-08-16T02:28:20.084523+0000","last_peered":"2024-08-16T02:28:20.084523+0000","last_clean":"2024-08-16T02:28:20.084523+0000","last_became_active":"2024-08-16T02:19:04.856804+0000","last_became_peered":"2024-08-16T02:19:04.856804+0000","last_unstale":"2024-08-16T02:28:20.084523+0000","last_undegraded":"2024-08-16T02:28:20.084523+0000","last_fullsized":"2024-08-16T02:28:20.084523+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.078748+0000","last_change":"2024-08-16T02:19:06.727198+0000","last_active":"2024-08-16T02:28:20.078748+0000","last_peered":"2024-08-16T02:28:20.078748+0000","last_clean":"2024-08-16T02:28:20.078748+0000","last_became_active":"2024-08-16T02:19:04.811355+0000","last_became_peered":"2024-08-16T02:19:04.811355+0000","last_unstale":"2024-08-16T02:28:20.078748+0000","last_undegraded":"2024-08-16T02:28:20.078748+0000","last_fullsized":"2024-08-16T02:28:20.078748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.078358+0000","last_change":"2024-08-16T02:19:06.727430+0000","last_active":"2024-08-16T02:28:20.078358+0000","last_peered":"2024-08-16T02:28:20.078358+0000","last_clean":"2024-08-16T02:28:20.078358+0000","last_became_active":"2024-08-16T02:19:04.829841+0000","last_became_peered":"2024-08-16T02:19:04.829841+0000","last_unstale":"2024-08-16T02:28:20.078358+0000","last_undegraded":"2024-08-16T02:28:20.078358+0000","last_fullsized":"2024-08-16T02:28:20.078358+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":326,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.168769+0000","last_change":"2024-08-16T02:19:06.726589+0000","last_active":"2024-08-16T02:28:20.168769+0000","last_peered":"2024-08-16T02:28:20.168769+0000","last_clean":"2024-08-16T02:28:20.168769+0000","last_became_active":"2024-08-16T02:19:04.859597+0000","last_became_peered":"2024-08-16T02:19:04.859597+0000","last_unstale":"2024-08-16T02:28:20.168769+0000","last_undegraded":"2024-08-16T02:28:20.168769+0000","last_fullsized":"2024-08-16T02:28:20.168769+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:21.110210+0000","last_change":"2024-08-16T02:19:06.731444+0000","last_active":"2024-08-16T02:28:21.110210+0000","last_peered":"2024-08-16T02:28:21.110210+0000","last_clean":"2024-08-16T02:28:21.110210+0000","last_became_active":"2024-08-16T02:19:04.864594+0000","last_became_peered":"2024-08-16T02:19:04.864594+0000","last_unstale":"2024-08-16T02:28:21.110210+0000","last_undegraded":"2024-08-16T02:28:21.110210+0000","last_fullsized":"2024-08-16T02:28:21.110210+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:21.066105+0000","last_change":"2024-08-16T02:19:06.726645+0000","last_active":"2024-08-16T02:28:21.066105+0000","last_peered":"2024-08-16T02:28:21.066105+0000","last_clean":"2024-08-16T02:28:21.066105+0000","last_became_active":"2024-08-16T02:19:04.825684+0000","last_became_peered":"2024-08-16T02:19:04.825684+0000","last_unstale":"2024-08-16T02:28:21.066105+0000","last_undegraded":"2024-08-16T02:28:21.066105+0000","last_fullsized":"2024-08-16T02:28:21.066105+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":325,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.093643+0000","last_change":"2024-08-16T02:19:06.731349+0000","last_active":"2024-08-16T02:28:20.093643+0000","last_peered":"2024-08-16T02:28:20.093643+0000","last_clean":"2024-08-16T02:28:20.093643+0000","last_became_active":"2024-08-16T02:19:04.824906+0000","last_became_peered":"2024-08-16T02:19:04.824906+0000","last_unstale":"2024-08-16T02:28:20.093643+0000","last_undegraded":"2024-08-16T02:28:20.093643+0000","last_fullsized":"2024-08-16T02:28:20.093643+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":393,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.084740+0000","last_change":"2024-08-16T02:19:03.779526+0000","last_active":"2024-08-16T02:28:20.084740+0000","last_peered":"2024-08-16T02:28:20.084740+0000","last_clean":"2024-08-16T02:28:20.084740+0000","last_became_active":"2024-08-16T02:19:03.745241+0000","last_became_peered":"2024-08-16T02:19:03.745241+0000","last_unstale":"2024-08-16T02:28:20.084740+0000","last_undegraded":"2024-08-16T02:28:20.084740+0000","last_fullsized":"2024-08-16T02:28:20.084740+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_clean_scrub_stamp":"2024-08-16T02:19:02.675182+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160288038912,"internally_reserved":0,"allocated":370581504,"data_stored":370581504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":98168,"kb_used_data":98168,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52199560,"statfs":{"total":53552873472,"available":53452349440,"internally_reserved":0,"allocated":100524032,"data_stored":100524032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":150032,"kb_used_data":150032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52147696,"statfs":{"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":114432,"kb_used_data":114432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52183296,"statfs":{"total":53552873472,"available":53435695104,"internally_reserved":0,"allocated":117178368,"data_stored":117178368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53436039168,"internally_reserved":0,"allocated":116834304,"data_stored":116834304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53452759040,"internally_reserved":0,"allocated":100114432,"data_stored":100114432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:25.924 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:26.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:26.027+0000 7f1f08952640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:26.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:26.037+0000 7f1f08952640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:26.229 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:28:26.229 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-08-16T02:28:26.242 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-08-16T02:28:24.771991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":362632,"kb_used_data":362632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156530552,"statfs":{"total":160658620416,"available":160287285248,"internally_reserved":0,"allocated":371335168,"data_stored":371335168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168070818,"num_objects":-221,"num_object_clones":0,"num_object_copies":-442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28112,"num_read_kb":-145426,"num_write":-1526,"num_write_kb":-203274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.759186"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.079643+0000","last_change":"2024-08-16T02:19:06.726892+0000","last_active":"2024-08-16T02:28:20.079643+0000","last_peered":"2024-08-16T02:28:20.079643+0000","last_clean":"2024-08-16T02:28:20.079643+0000","last_became_active":"2024-08-16T02:19:04.800574+0000","last_became_peered":"2024-08-16T02:19:04.800574+0000","last_unstale":"2024-08-16T02:28:20.079643+0000","last_undegraded":"2024-08-16T02:28:20.079643+0000","last_fullsized":"2024-08-16T02:28:20.079643+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.084523+0000","last_change":"2024-08-16T02:19:06.731654+0000","last_active":"2024-08-16T02:28:20.084523+0000","last_peered":"2024-08-16T02:28:20.084523+0000","last_clean":"2024-08-16T02:28:20.084523+0000","last_became_active":"2024-08-16T02:19:04.856804+0000","last_became_peered":"2024-08-16T02:19:04.856804+0000","last_unstale":"2024-08-16T02:28:20.084523+0000","last_undegraded":"2024-08-16T02:28:20.084523+0000","last_fullsized":"2024-08-16T02:28:20.084523+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.078748+0000","last_change":"2024-08-16T02:19:06.727198+0000","last_active":"2024-08-16T02:28:20.078748+0000","last_peered":"2024-08-16T02:28:20.078748+0000","last_clean":"2024-08-16T02:28:20.078748+0000","last_became_active":"2024-08-16T02:19:04.811355+0000","last_became_peered":"2024-08-16T02:19:04.811355+0000","last_unstale":"2024-08-16T02:28:20.078748+0000","last_undegraded":"2024-08-16T02:28:20.078748+0000","last_fullsized":"2024-08-16T02:28:20.078748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.078358+0000","last_change":"2024-08-16T02:19:06.727430+0000","last_active":"2024-08-16T02:28:20.078358+0000","last_peered":"2024-08-16T02:28:20.078358+0000","last_clean":"2024-08-16T02:28:20.078358+0000","last_became_active":"2024-08-16T02:19:04.829841+0000","last_became_peered":"2024-08-16T02:19:04.829841+0000","last_unstale":"2024-08-16T02:28:20.078358+0000","last_undegraded":"2024-08-16T02:28:20.078358+0000","last_fullsized":"2024-08-16T02:28:20.078358+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":326,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.168769+0000","last_change":"2024-08-16T02:19:06.726589+0000","last_active":"2024-08-16T02:28:20.168769+0000","last_peered":"2024-08-16T02:28:20.168769+0000","last_clean":"2024-08-16T02:28:20.168769+0000","last_became_active":"2024-08-16T02:19:04.859597+0000","last_became_peered":"2024-08-16T02:19:04.859597+0000","last_unstale":"2024-08-16T02:28:20.168769+0000","last_undegraded":"2024-08-16T02:28:20.168769+0000","last_fullsized":"2024-08-16T02:28:20.168769+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:21.110210+0000","last_change":"2024-08-16T02:19:06.731444+0000","last_active":"2024-08-16T02:28:21.110210+0000","last_peered":"2024-08-16T02:28:21.110210+0000","last_clean":"2024-08-16T02:28:21.110210+0000","last_became_active":"2024-08-16T02:19:04.864594+0000","last_became_peered":"2024-08-16T02:19:04.864594+0000","last_unstale":"2024-08-16T02:28:21.110210+0000","last_undegraded":"2024-08-16T02:28:21.110210+0000","last_fullsized":"2024-08-16T02:28:21.110210+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:21.066105+0000","last_change":"2024-08-16T02:19:06.726645+0000","last_active":"2024-08-16T02:28:21.066105+0000","last_peered":"2024-08-16T02:28:21.066105+0000","last_clean":"2024-08-16T02:28:21.066105+0000","last_became_active":"2024-08-16T02:19:04.825684+0000","last_became_peered":"2024-08-16T02:19:04.825684+0000","last_unstale":"2024-08-16T02:28:21.066105+0000","last_undegraded":"2024-08-16T02:28:21.066105+0000","last_fullsized":"2024-08-16T02:28:21.066105+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":325,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.093643+0000","last_change":"2024-08-16T02:19:06.731349+0000","last_active":"2024-08-16T02:28:20.093643+0000","last_peered":"2024-08-16T02:28:20.093643+0000","last_clean":"2024-08-16T02:28:20.093643+0000","last_became_active":"2024-08-16T02:19:04.824906+0000","last_became_peered":"2024-08-16T02:19:04.824906+0000","last_unstale":"2024-08-16T02:28:20.093643+0000","last_undegraded":"2024-08-16T02:28:20.093643+0000","last_fullsized":"2024-08-16T02:28:20.093643+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":393,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.084740+0000","last_change":"2024-08-16T02:19:03.779526+0000","last_active":"2024-08-16T02:28:20.084740+0000","last_peered":"2024-08-16T02:28:20.084740+0000","last_clean":"2024-08-16T02:28:20.084740+0000","last_became_active":"2024-08-16T02:19:03.745241+0000","last_became_peered":"2024-08-16T02:19:03.745241+0000","last_unstale":"2024-08-16T02:28:20.084740+0000","last_undegraded":"2024-08-16T02:28:20.084740+0000","last_fullsized":"2024-08-16T02:28:20.084740+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_clean_scrub_stamp":"2024-08-16T02:19:02.675182+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160288038912,"internally_reserved":0,"allocated":370581504,"data_stored":370581504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":98168,"kb_used_data":98168,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52199560,"statfs":{"total":53552873472,"available":53452349440,"internally_reserved":0,"allocated":100524032,"data_stored":100524032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":150032,"kb_used_data":150032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52147696,"statfs":{"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":114432,"kb_used_data":114432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52183296,"statfs":{"total":53552873472,"available":53435695104,"internally_reserved":0,"allocated":117178368,"data_stored":117178368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53436039168,"internally_reserved":0,"allocated":116834304,"data_stored":116834304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53452759040,"internally_reserved":0,"allocated":100114432,"data_stored":100114432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:26.243 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-16T02:28:26.244 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:26.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:26.346+0000 7f5b34cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:26.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:26.354+0000 7f5b34cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:26.546 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:28:26.546 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-08-16T02:28:26.559 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-08-16T02:28:24.771991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":362632,"kb_used_data":362632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156530552,"statfs":{"total":160658620416,"available":160287285248,"internally_reserved":0,"allocated":371335168,"data_stored":371335168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168070818,"num_objects":-221,"num_object_clones":0,"num_object_copies":-442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28112,"num_read_kb":-145426,"num_write":-1526,"num_write_kb":-203274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.759186"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.079643+0000","last_change":"2024-08-16T02:19:06.726892+0000","last_active":"2024-08-16T02:28:20.079643+0000","last_peered":"2024-08-16T02:28:20.079643+0000","last_clean":"2024-08-16T02:28:20.079643+0000","last_became_active":"2024-08-16T02:19:04.800574+0000","last_became_peered":"2024-08-16T02:19:04.800574+0000","last_unstale":"2024-08-16T02:28:20.079643+0000","last_undegraded":"2024-08-16T02:28:20.079643+0000","last_fullsized":"2024-08-16T02:28:20.079643+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.084523+0000","last_change":"2024-08-16T02:19:06.731654+0000","last_active":"2024-08-16T02:28:20.084523+0000","last_peered":"2024-08-16T02:28:20.084523+0000","last_clean":"2024-08-16T02:28:20.084523+0000","last_became_active":"2024-08-16T02:19:04.856804+0000","last_became_peered":"2024-08-16T02:19:04.856804+0000","last_unstale":"2024-08-16T02:28:20.084523+0000","last_undegraded":"2024-08-16T02:28:20.084523+0000","last_fullsized":"2024-08-16T02:28:20.084523+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.078748+0000","last_change":"2024-08-16T02:19:06.727198+0000","last_active":"2024-08-16T02:28:20.078748+0000","last_peered":"2024-08-16T02:28:20.078748+0000","last_clean":"2024-08-16T02:28:20.078748+0000","last_became_active":"2024-08-16T02:19:04.811355+0000","last_became_peered":"2024-08-16T02:19:04.811355+0000","last_unstale":"2024-08-16T02:28:20.078748+0000","last_undegraded":"2024-08-16T02:28:20.078748+0000","last_fullsized":"2024-08-16T02:28:20.078748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":324,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.078358+0000","last_change":"2024-08-16T02:19:06.727430+0000","last_active":"2024-08-16T02:28:20.078358+0000","last_peered":"2024-08-16T02:28:20.078358+0000","last_clean":"2024-08-16T02:28:20.078358+0000","last_became_active":"2024-08-16T02:19:04.829841+0000","last_became_peered":"2024-08-16T02:19:04.829841+0000","last_unstale":"2024-08-16T02:28:20.078358+0000","last_undegraded":"2024-08-16T02:28:20.078358+0000","last_fullsized":"2024-08-16T02:28:20.078358+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":326,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.168769+0000","last_change":"2024-08-16T02:19:06.726589+0000","last_active":"2024-08-16T02:28:20.168769+0000","last_peered":"2024-08-16T02:28:20.168769+0000","last_clean":"2024-08-16T02:28:20.168769+0000","last_became_active":"2024-08-16T02:19:04.859597+0000","last_became_peered":"2024-08-16T02:19:04.859597+0000","last_unstale":"2024-08-16T02:28:20.168769+0000","last_undegraded":"2024-08-16T02:28:20.168769+0000","last_fullsized":"2024-08-16T02:28:20.168769+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":327,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:21.110210+0000","last_change":"2024-08-16T02:19:06.731444+0000","last_active":"2024-08-16T02:28:21.110210+0000","last_peered":"2024-08-16T02:28:21.110210+0000","last_clean":"2024-08-16T02:28:21.110210+0000","last_became_active":"2024-08-16T02:19:04.864594+0000","last_became_peered":"2024-08-16T02:19:04.864594+0000","last_unstale":"2024-08-16T02:28:21.110210+0000","last_undegraded":"2024-08-16T02:28:21.110210+0000","last_fullsized":"2024-08-16T02:28:21.110210+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":327,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:21.066105+0000","last_change":"2024-08-16T02:19:06.726645+0000","last_active":"2024-08-16T02:28:21.066105+0000","last_peered":"2024-08-16T02:28:21.066105+0000","last_clean":"2024-08-16T02:28:21.066105+0000","last_became_active":"2024-08-16T02:19:04.825684+0000","last_became_peered":"2024-08-16T02:19:04.825684+0000","last_unstale":"2024-08-16T02:28:21.066105+0000","last_undegraded":"2024-08-16T02:28:21.066105+0000","last_fullsized":"2024-08-16T02:28:21.066105+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":325,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.093643+0000","last_change":"2024-08-16T02:19:06.731349+0000","last_active":"2024-08-16T02:28:20.093643+0000","last_peered":"2024-08-16T02:28:20.093643+0000","last_clean":"2024-08-16T02:28:20.093643+0000","last_became_active":"2024-08-16T02:19:04.824906+0000","last_became_peered":"2024-08-16T02:19:04.824906+0000","last_unstale":"2024-08-16T02:28:20.093643+0000","last_undegraded":"2024-08-16T02:28:20.093643+0000","last_fullsized":"2024-08-16T02:28:20.093643+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":393,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-08-16T02:28:20.084740+0000","last_change":"2024-08-16T02:19:03.779526+0000","last_active":"2024-08-16T02:28:20.084740+0000","last_peered":"2024-08-16T02:28:20.084740+0000","last_clean":"2024-08-16T02:28:20.084740+0000","last_became_active":"2024-08-16T02:19:03.745241+0000","last_became_peered":"2024-08-16T02:19:03.745241+0000","last_unstale":"2024-08-16T02:28:20.084740+0000","last_undegraded":"2024-08-16T02:28:20.084740+0000","last_fullsized":"2024-08-16T02:28:20.084740+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_clean_scrub_stamp":"2024-08-16T02:19:02.675182+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160288038912,"internally_reserved":0,"allocated":370581504,"data_stored":370581504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":98168,"kb_used_data":98168,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52199560,"statfs":{"total":53552873472,"available":53452349440,"internally_reserved":0,"allocated":100524032,"data_stored":100524032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":150032,"kb_used_data":150032,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52147696,"statfs":{"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":114432,"kb_used_data":114432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52183296,"statfs":{"total":53552873472,"available":53435695104,"internally_reserved":0,"allocated":117178368,"data_stored":117178368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53436039168,"internally_reserved":0,"allocated":116834304,"data_stored":116834304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53399240704,"internally_reserved":0,"allocated":153632768,"data_stored":153632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53452759040,"internally_reserved":0,"allocated":100114432,"data_stored":100114432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:26.559 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:28:26.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:26.662+0000 7fc4ccc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:26.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:26.669+0000 7fc4ccc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:26.869 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:28:26.869 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":119,"fsid":"a2704e77-df6d-4e59-a0c8-3375fbc7352b","created":"2024-08-16T02:18:53.821789+0000","modified":"2024-08-16T02:28:22.019115+0000","last_up_change":"2024-08-16T02:19:01.666431+0000","last_in_change":"2024-08-16T02:18:55.722536+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:19:02.630075+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:19:03.622648+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8de1315e-0175-4dc6-b305-fb3198ae930f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":102364345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":102364345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":102364345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":102364345}]},"public_addr":"172.21.15.87:6805/102364345","cluster_addr":"172.21.15.87:6802/102364345","heartbeat_back_addr":"172.21.15.87:6811/102364345","heartbeat_front_addr":"172.21.15.87:6810/102364345","state":["exists","up"]},{"osd":1,"uuid":"169faddf-7bed-4cf1-9d94-92689aff3df9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1247754176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1247754176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":1247754176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1247754176}]},"public_addr":"172.21.15.87:6804/1247754176","cluster_addr":"172.21.15.87:6803/1247754176","heartbeat_back_addr":"172.21.15.87:6807/1247754176","heartbeat_front_addr":"172.21.15.87:6808/1247754176","state":["exists","up"]},{"osd":2,"uuid":"e18ba384-d888-4ff8-89e1-3b1799a1a55d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":1342115893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":1342115893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":1342115893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":1342115893}]},"public_addr":"172.21.15.87:6809/1342115893","cluster_addr":"172.21.15.87:6812/1342115893","heartbeat_back_addr":"172.21.15.87:6813/1342115893","heartbeat_front_addr":"172.21.15.87:6806/1342115893","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":13},{"begin":17,"length":1},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":28,"length":8}]},{"pool":4,"snaps":[{"begin":2,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:28:27.883 INFO:tasks.ceph:Scrubbing osd.0 2024-08-16T02:28:27.883 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:28:28.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:27.993+0000 7f90d2064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.005+0000 7f90d2064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.020 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-08-16T02:28:28.021 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "" 2024-08-16T02:28:28.021 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-08-16T02:28:28.032 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-16T02:28:28.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.136+0000 7fd477392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.144+0000 7fd477392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.338 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 0 to deep-scrub 2024-08-16T02:28:28.355 INFO:tasks.ceph:Scrubbing osd.1 2024-08-16T02:28:28.356 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:28:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.458+0000 7f0e7a064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.470+0000 7f0e7a064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.484 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-08-16T02:28:28.484 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "" 2024-08-16T02:28:28.485 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-08-16T02:28:28.495 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-16T02:28:28.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.602+0000 7f9440192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.610+0000 7f9440192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.809 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 1 to deep-scrub 2024-08-16T02:28:28.825 INFO:tasks.ceph:Scrubbing osd.2 2024-08-16T02:28:28.826 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:28:28.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.934+0000 7f9db2349640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:28.946+0000 7f9db2349640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:28.960 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-08-16T02:28:28.960 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "" 2024-08-16T02:28:28.960 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-08-16T02:28:28.971 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-16T02:28:29.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:29.075+0000 7f6ec666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:29.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:29.084+0000 7f6ec666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:29.279 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 2 to deep-scrub 2024-08-16T02:28:29.298 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:29.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:29.412+0000 7f30e4992640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:29.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:29.420+0000 7f30e4992640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:29.611 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:28:29.611 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-08-16T02:28:29.624 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-08-16T02:28:28.772644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":30932,"kb_used_data":30932,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156862252,"statfs":{"total":160658620416,"available":160626946048,"internally_reserved":0,"allocated":31674368,"data_stored":31674368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168070810,"num_objects":-220,"num_object_clones":0,"num_object_copies":-440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28100,"num_read_kb":-145418,"num_write":-1510,"num_write_kb":-203262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001461"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:22.106298+0000","last_change":"2024-08-16T02:19:06.726892+0000","last_active":"2024-08-16T02:28:22.106298+0000","last_peered":"2024-08-16T02:28:22.106298+0000","last_clean":"2024-08-16T02:28:22.106298+0000","last_became_active":"2024-08-16T02:19:04.800574+0000","last_became_peered":"2024-08-16T02:19:04.800574+0000","last_unstale":"2024-08-16T02:28:22.106298+0000","last_undegraded":"2024-08-16T02:28:22.106298+0000","last_fullsized":"2024-08-16T02:28:22.106298+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:22.072102+0000","last_change":"2024-08-16T02:19:06.731654+0000","last_active":"2024-08-16T02:28:22.072102+0000","last_peered":"2024-08-16T02:28:22.072102+0000","last_clean":"2024-08-16T02:28:22.072102+0000","last_became_active":"2024-08-16T02:19:04.856804+0000","last_became_peered":"2024-08-16T02:19:04.856804+0000","last_unstale":"2024-08-16T02:28:22.072102+0000","last_undegraded":"2024-08-16T02:28:22.072102+0000","last_fullsized":"2024-08-16T02:28:22.072102+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:22.085323+0000","last_change":"2024-08-16T02:19:06.727198+0000","last_active":"2024-08-16T02:28:22.085323+0000","last_peered":"2024-08-16T02:28:22.085323+0000","last_clean":"2024-08-16T02:28:22.085323+0000","last_became_active":"2024-08-16T02:19:04.811355+0000","last_became_peered":"2024-08-16T02:19:04.811355+0000","last_unstale":"2024-08-16T02:28:22.085323+0000","last_undegraded":"2024-08-16T02:28:22.085323+0000","last_fullsized":"2024-08-16T02:28:22.085323+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:22.072384+0000","last_change":"2024-08-16T02:19:06.727430+0000","last_active":"2024-08-16T02:28:22.072384+0000","last_peered":"2024-08-16T02:28:22.072384+0000","last_clean":"2024-08-16T02:28:22.072384+0000","last_became_active":"2024-08-16T02:19:04.829841+0000","last_became_peered":"2024-08-16T02:19:04.829841+0000","last_unstale":"2024-08-16T02:28:22.072384+0000","last_undegraded":"2024-08-16T02:28:22.072384+0000","last_fullsized":"2024-08-16T02:28:22.072384+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:22.697137+0000","last_change":"2024-08-16T02:19:06.726589+0000","last_active":"2024-08-16T02:28:22.697137+0000","last_peered":"2024-08-16T02:28:22.697137+0000","last_clean":"2024-08-16T02:28:22.697137+0000","last_became_active":"2024-08-16T02:19:04.859597+0000","last_became_peered":"2024-08-16T02:19:04.859597+0000","last_unstale":"2024-08-16T02:28:22.697137+0000","last_undegraded":"2024-08-16T02:28:22.697137+0000","last_fullsized":"2024-08-16T02:28:22.697137+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:23.070939+0000","last_change":"2024-08-16T02:19:06.731444+0000","last_active":"2024-08-16T02:28:23.070939+0000","last_peered":"2024-08-16T02:28:23.070939+0000","last_clean":"2024-08-16T02:28:23.070939+0000","last_became_active":"2024-08-16T02:19:04.864594+0000","last_became_peered":"2024-08-16T02:19:04.864594+0000","last_unstale":"2024-08-16T02:28:23.070939+0000","last_undegraded":"2024-08-16T02:28:23.070939+0000","last_fullsized":"2024-08-16T02:28:23.070939+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:22.833568+0000","last_change":"2024-08-16T02:19:06.726645+0000","last_active":"2024-08-16T02:28:22.833568+0000","last_peered":"2024-08-16T02:28:22.833568+0000","last_clean":"2024-08-16T02:28:22.833568+0000","last_became_active":"2024-08-16T02:19:04.825684+0000","last_became_peered":"2024-08-16T02:19:04.825684+0000","last_unstale":"2024-08-16T02:28:22.833568+0000","last_undegraded":"2024-08-16T02:28:22.833568+0000","last_fullsized":"2024-08-16T02:28:22.833568+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":331,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:22.119272+0000","last_change":"2024-08-16T02:19:06.731349+0000","last_active":"2024-08-16T02:28:22.119272+0000","last_peered":"2024-08-16T02:28:22.119272+0000","last_clean":"2024-08-16T02:28:22.119272+0000","last_became_active":"2024-08-16T02:19:04.824906+0000","last_became_peered":"2024-08-16T02:19:04.824906+0000","last_unstale":"2024-08-16T02:28:22.119272+0000","last_undegraded":"2024-08-16T02:28:22.119272+0000","last_fullsized":"2024-08-16T02:28:22.119272+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:03.681537+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":399,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:22.077273+0000","last_change":"2024-08-16T02:19:03.779526+0000","last_active":"2024-08-16T02:28:22.077273+0000","last_peered":"2024-08-16T02:28:22.077273+0000","last_clean":"2024-08-16T02:28:22.077273+0000","last_became_active":"2024-08-16T02:19:03.745241+0000","last_became_peered":"2024-08-16T02:19:03.745241+0000","last_unstale":"2024-08-16T02:28:22.077273+0000","last_undegraded":"2024-08-16T02:28:22.077273+0000","last_fullsized":"2024-08-16T02:28:22.077273+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:02.675182+0000","last_clean_scrub_stamp":"2024-08-16T02:19:02.675182+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653049856,"internally_reserved":0,"allocated":5570560,"data_stored":5570560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550522368,"internally_reserved":0,"allocated":2351104,"data_stored":2351104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706227,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1120,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296608,"statfs":{"total":53552873472,"available":53551726592,"internally_reserved":0,"allocated":1146880,"data_stored":1146880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2296,"kb_used_data":2296,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295432,"statfs":{"total":53552873472,"available":53550522368,"internally_reserved":0,"allocated":2351104,"data_stored":2351104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738931,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":27516,"kb_used_data":27516,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52270212,"statfs":{"total":53552873472,"available":53524697088,"internally_reserved":0,"allocated":28176384,"data_stored":28176384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550522368,"internally_reserved":0,"allocated":2351104,"data_stored":2351104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550800896,"internally_reserved":0,"allocated":2072576,"data_stored":2072576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550522368,"internally_reserved":0,"allocated":2351104,"data_stored":2351104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551726592,"internally_reserved":0,"allocated":1146880,"data_stored":1146880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:29.626 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-16T02:19:03.681537+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=3, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.627 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-16T02:19:03.681537+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=3, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.627 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-16T02:19:03.681537+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=3, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.627 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-16T02:19:03.681537+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=3, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.627 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-16T02:19:03.681537+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=3, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.627 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-16T02:19:03.681537+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=3, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.628 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-16T02:19:03.681537+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=3, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.628 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-16T02:19:03.681537+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=3, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.628 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-16T02:19:02.675182+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=2, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=26, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:29.628 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-16T02:28:49.629 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:49.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:49.747+0000 7f17e6c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:49.754+0000 7f17e6c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:49.950 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-08-16T02:28:49.951 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-08-16T02:28:49.963 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-08-16T02:28:48.777731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5440,"kb_used_data":5440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156887744,"statfs":{"total":160658620416,"available":160653049856,"internally_reserved":0,"allocated":5570560,"data_stored":5570560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003186"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:28.815553+0000","last_change":"2024-08-16T02:28:28.850754+0000","last_active":"2024-08-16T02:28:28.815553+0000","last_peered":"2024-08-16T02:28:28.815553+0000","last_clean":"2024-08-16T02:28:28.815553+0000","last_became_active":"2024-08-16T02:19:04.800574+0000","last_became_peered":"2024-08-16T02:19:04.800574+0000","last_unstale":"2024-08-16T02:28:28.815553+0000","last_undegraded":"2024-08-16T02:28:28.815553+0000","last_fullsized":"2024-08-16T02:28:28.815553+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:28.850586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:28.850586+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:28.816127+0000","last_change":"2024-08-16T02:28:28.836380+0000","last_active":"2024-08-16T02:28:28.816127+0000","last_peered":"2024-08-16T02:28:28.816127+0000","last_clean":"2024-08-16T02:28:28.816127+0000","last_became_active":"2024-08-16T02:19:04.856804+0000","last_became_peered":"2024-08-16T02:19:04.856804+0000","last_unstale":"2024-08-16T02:28:28.816127+0000","last_undegraded":"2024-08-16T02:28:28.816127+0000","last_fullsized":"2024-08-16T02:28:28.816127+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:28.836215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:28.836215+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:28.813973+0000","last_change":"2024-08-16T02:28:28.848172+0000","last_active":"2024-08-16T02:28:28.813973+0000","last_peered":"2024-08-16T02:28:28.813973+0000","last_clean":"2024-08-16T02:28:28.813973+0000","last_became_active":"2024-08-16T02:19:04.811355+0000","last_became_peered":"2024-08-16T02:19:04.811355+0000","last_unstale":"2024-08-16T02:28:28.813973+0000","last_undegraded":"2024-08-16T02:28:28.813973+0000","last_fullsized":"2024-08-16T02:28:28.813973+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:28.848000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:28.848000+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:28.812422+0000","last_change":"2024-08-16T02:28:28.846150+0000","last_active":"2024-08-16T02:28:28.812422+0000","last_peered":"2024-08-16T02:28:28.812422+0000","last_clean":"2024-08-16T02:28:28.812422+0000","last_became_active":"2024-08-16T02:19:04.829841+0000","last_became_peered":"2024-08-16T02:19:04.829841+0000","last_unstale":"2024-08-16T02:28:28.812422+0000","last_undegraded":"2024-08-16T02:28:28.812422+0000","last_fullsized":"2024-08-16T02:28:28.812422+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:28.845961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:28.845961+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":334,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:28.340049+0000","last_change":"2024-08-16T02:28:28.358460+0000","last_active":"2024-08-16T02:28:28.340049+0000","last_peered":"2024-08-16T02:28:28.340049+0000","last_clean":"2024-08-16T02:28:28.340049+0000","last_became_active":"2024-08-16T02:19:04.859597+0000","last_became_peered":"2024-08-16T02:19:04.859597+0000","last_unstale":"2024-08-16T02:28:28.340049+0000","last_undegraded":"2024-08-16T02:28:28.340049+0000","last_fullsized":"2024-08-16T02:28:28.340049+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2024-08-16T02:28:28.358286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:28.358286+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:29.284433+0000","last_change":"2024-08-16T02:28:29.298546+0000","last_active":"2024-08-16T02:28:29.284433+0000","last_peered":"2024-08-16T02:28:29.284433+0000","last_clean":"2024-08-16T02:28:29.284433+0000","last_became_active":"2024-08-16T02:19:04.864594+0000","last_became_peered":"2024-08-16T02:19:04.864594+0000","last_unstale":"2024-08-16T02:28:29.284433+0000","last_undegraded":"2024-08-16T02:28:29.284433+0000","last_fullsized":"2024-08-16T02:28:29.284433+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:29.298309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:29.298309+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:29.282231+0000","last_change":"2024-08-16T02:28:29.296428+0000","last_active":"2024-08-16T02:28:29.282231+0000","last_peered":"2024-08-16T02:28:29.282231+0000","last_clean":"2024-08-16T02:28:29.282231+0000","last_became_active":"2024-08-16T02:19:04.825684+0000","last_became_peered":"2024-08-16T02:19:04.825684+0000","last_unstale":"2024-08-16T02:28:29.282231+0000","last_undegraded":"2024-08-16T02:28:29.282231+0000","last_fullsized":"2024-08-16T02:28:29.282231+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:29.296245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:29.296245+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:28.811540+0000","last_change":"2024-08-16T02:28:28.838120+0000","last_active":"2024-08-16T02:28:28.811540+0000","last_peered":"2024-08-16T02:28:28.811540+0000","last_clean":"2024-08-16T02:28:28.811540+0000","last_became_active":"2024-08-16T02:19:04.824906+0000","last_became_peered":"2024-08-16T02:19:04.824906+0000","last_unstale":"2024-08-16T02:28:28.811540+0000","last_undegraded":"2024-08-16T02:28:28.811540+0000","last_fullsized":"2024-08-16T02:28:28.811540+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2024-08-16T02:28:28.837908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:28.837908+0000","last_clean_scrub_stamp":"2024-08-16T02:19:03.681537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":401,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:28.811548+0000","last_change":"2024-08-16T02:28:28.866638+0000","last_active":"2024-08-16T02:28:28.811548+0000","last_peered":"2024-08-16T02:28:28.811548+0000","last_clean":"2024-08-16T02:28:28.811548+0000","last_became_active":"2024-08-16T02:19:03.745241+0000","last_became_peered":"2024-08-16T02:19:03.745241+0000","last_unstale":"2024-08-16T02:28:28.811548+0000","last_undegraded":"2024-08-16T02:28:28.811548+0000","last_fullsized":"2024-08-16T02:28:28.811548+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-08-16T02:28:28.866449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:28.866449+0000","last_clean_scrub_stamp":"2024-08-16T02:19:02.675182+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653049856,"internally_reserved":0,"allocated":5570560,"data_stored":5570560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550522368,"internally_reserved":0,"allocated":2351104,"data_stored":2351104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1120,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296608,"statfs":{"total":53552873472,"available":53551726592,"internally_reserved":0,"allocated":1146880,"data_stored":1146880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2296,"kb_used_data":2296,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295432,"statfs":{"total":53552873472,"available":53550522368,"internally_reserved":0,"allocated":2351104,"data_stored":2351104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738951,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2024,"kb_used_data":2024,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295704,"statfs":{"total":53552873472,"available":53550800896,"internally_reserved":0,"allocated":2072576,"data_stored":2072576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550522368,"internally_reserved":0,"allocated":2351104,"data_stored":2351104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550800896,"internally_reserved":0,"allocated":2072576,"data_stored":2072576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550522368,"internally_reserved":0,"allocated":2351104,"data_stored":2351104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551726592,"internally_reserved":0,"allocated":1146880,"data_stored":1146880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:49.965 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-16T02:28:50.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:50.065+0000 7f2bbc592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:50.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-08-16T02:28:50.076+0000 7f2bbc592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:50.298 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-16T02:28:50.298 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-16T02:28:50.298 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-16T02:28:50.298 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:28:50.379 INFO:tasks.ceph.osd.0:Stopped 2024-08-16T02:28:50.380 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-16T02:28:50.380 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:28:50.455 INFO:tasks.ceph.osd.1:Stopped 2024-08-16T02:28:50.455 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-16T02:28:50.456 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:28:50.529 INFO:tasks.ceph.osd.2:Stopped 2024-08-16T02:28:50.530 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-16T02:28:50.530 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-16T02:28:50.530 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:28:50.558 INFO:tasks.ceph.mgr.x:Stopped 2024-08-16T02:28:50.559 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-16T02:28:50.559 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-16T02:28:50.559 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:28:50.612 INFO:tasks.ceph.mon.a:Stopped 2024-08-16T02:28:50.612 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-16T02:28:50.612 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(POOL_FULL\)' | head -n 1 2024-08-16T02:28:50.681 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:28:50.681 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-16T02:28:50.821 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:28:50.821 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-16T02:28:50.949 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:28:50.949 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-16T02:28:51.007 INFO:tasks.ceph:Archiving mon data... 2024-08-16T02:28:51.009 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857403/data/mon.a.tgz 2024-08-16T02:28:51.009 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-08-16T02:28:51.023 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.DGjI6EzEWE 2024-08-16T02:28:51.024 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.DGjI6EzEWE 2024-08-16T02:28:51.154 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.DGjI6EzEWE 2024-08-16T02:28:51.236 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.DGjI6EzEWE is 217KB 2024-08-16T02:28:51.328 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.DGjI6EzEWE 2024-08-16T02:28:51.343 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-16T02:28:51.343 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-16T02:28:51.462 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-16T02:28:51.462 INFO:tasks.ceph:Archiving crash dumps... 2024-08-16T02:28:51.464 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857403/remote/smithi087/crash 2024-08-16T02:28:51.465 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-16T02:28:51.526 INFO:tasks.ceph:Compressing logs... 2024-08-16T02:28:51.526 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-16T02:28:51.607 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34069.log 2024-08-16T02:28:51.608 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31413.log 2024-08-16T02:28:51.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31623.log 2024-08-16T02:28:51.608 INFO:teuthology.orchestra.run.smithi087.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34069.log.gz 2024-08-16T02:28:51.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34237.log 2024-08-16T02:28:51.608 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31413.log.gz 2024-08-16T02:28:51.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34279.log 2024-08-16T02:28:51.609 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31623.log.gz 2024-08-16T02:28:51.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34237.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-16T02:28:51.609 INFO:teuthology.orchestra.run.smithi087.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34237.log.gz 2024-08-16T02:28:51.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34279.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-16T02:28:51.609 INFO:teuthology.orchestra.run.smithi087.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34279.log.gz 2024-08-16T02:28:51.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33943.log 2024-08-16T02:28:51.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34405.log 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33943.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi087.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33943.log.gz 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33985.log 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi087.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34405.log.gz 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.audit.log: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34321.log 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33985.log: 87.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31119.log 2024-08-16T02:28:51.611 INFO:teuthology.orchestra.run.smithi087.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33985.log.gz 2024-08-16T02:28:51.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31581.log 2024-08-16T02:28:51.611 INFO:teuthology.orchestra.run.smithi087.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34321.log.gz 2024-08-16T02:28:51.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34111.log 2024-08-16T02:28:51.611 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31119.log.gz 2024-08-16T02:28:51.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31842.log 2024-08-16T02:28:51.611 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31581.log.gz 2024-08-16T02:28:51.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34445.log 2024-08-16T02:28:51.612 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34111.log.gz 2024-08-16T02:28:51.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31842.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-16T02:28:51.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31049.log 2024-08-16T02:28:51.612 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34445.log.gz 2024-08-16T02:28:51.612 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31842.log.gz 2024-08-16T02:28:51.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-16T02:28:51.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31049.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-16T02:28:51.612 INFO:teuthology.orchestra.run.smithi087.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.31049.log.gz 2024-08-16T02:28:51.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31452.log 2024-08-16T02:28:51.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30601.log 2024-08-16T02:28:51.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31327.log 2024-08-16T02:28:51.613 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31452.log.gz 2024-08-16T02:28:51.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34195.log 2024-08-16T02:28:51.613 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30601.log.gz 2024-08-16T02:28:51.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31327.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-16T02:28:51.613 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31327.log.gz 2024-08-16T02:28:51.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34195.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29388.log 2024-08-16T02:28:51.614 INFO:teuthology.orchestra.run.smithi087.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.34195.log.gz 2024-08-16T02:28:51.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31245.log 2024-08-16T02:28:51.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.tmp-client.admin.29388.log: gzip -5 --verbose -- 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29388.log.gz 2024-08-16T02:28:51.614 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.34363.log 2024-08-16T02:28:51.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31245.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31245.log.gz 2024-08-16T02:28:51.614 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30968.log 2024-08-16T02:28:51.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34363.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34363.log.gz 2024-08-16T02:28:51.615 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31326.log 2024-08-16T02:28:51.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30968.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30968.log.gz 2024-08-16T02:28:51.615 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31453.log 2024-08-16T02:28:51.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31326.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31326.log.gz 2024-08-16T02:28:51.615 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30799.log 2024-08-16T02:28:51.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31453.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31453.log.gz 2024-08-16T02:28:51.615 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31024.log 2024-08-16T02:28:51.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30799.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31325.log 2024-08-16T02:28:51.616 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30799.log.gz 2024-08-16T02:28:51.616 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30561.log 2024-08-16T02:28:51.616 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31024.log.gz 2024-08-16T02:28:51.616 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.31325.log: /var/log/ceph/ceph-client.admin.31203.log 2024-08-16T02:28:51.616 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31325.log.gz 2024-08-16T02:28:51.616 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30561.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30561.log.gz 2024-08-16T02:28:51.616 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30440.log 2024-08-16T02:28:51.616 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30641.log 2024-08-16T02:28:51.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30440.log: /var/log/ceph/ceph-client.admin.31203.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31539.log 2024-08-16T02:28:51.617 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30440.log.gz 2024-08-16T02:28:51.617 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31203.log.gz 2024-08-16T02:28:51.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30641.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30641.log.gz 2024-08-16T02:28:51.617 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30829.log 2024-08-16T02:28:51.617 INFO:teuthology.orchestra.run.smithi087.stderr:gzip/var/log/ceph/ceph-client.admin.31539.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34153.log 2024-08-16T02:28:51.617 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31539.log.gz 2024-08-16T02:28:51.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30829.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30352.log 2024-08-16T02:28:51.618 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30829.log.gz 2024-08-16T02:28:51.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.34153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34027.log 2024-08-16T02:28:51.618 INFO:teuthology.orchestra.run.smithi087.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34153.log.gz 2024-08-16T02:28:51.618 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31161.log 2024-08-16T02:28:51.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.30352.log: /var/log/ceph/ceph-client.admin.34027.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30352.log.gz 2024-08-16T02:28:51.618 INFO:teuthology.orchestra.run.smithi087.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34027.log.gz 2024-08-16T02:28:51.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.31161.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31161.log.gz 2024-08-16T02:28:51.668 INFO:teuthology.orchestra.run.smithi087.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-16T02:28:51.821 INFO:teuthology.orchestra.run.smithi087.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-16T02:28:53.031 INFO:teuthology.orchestra.run.smithi087.stderr: 95.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-16T02:28:54.552 INFO:teuthology.orchestra.run.smithi087.stderr: 96.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-16T02:28:55.298 INFO:teuthology.orchestra.run.smithi087.stderr: 96.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-16T02:28:55.300 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-08-16T02:28:55.301 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m3.720s 2024-08-16T02:28:55.301 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m7.987s 2024-08-16T02:28:55.301 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m0.446s 2024-08-16T02:28:55.301 INFO:tasks.ceph:Archiving logs... 2024-08-16T02:28:55.302 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857403/remote/smithi087/log 2024-08-16T02:28:55.303 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-16T02:28:56.306 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-16T02:28:56.315 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-16T02:28:56.315 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-16T02:28:56.368 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-16T02:28:56.378 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-16T02:28:56.379 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:28:56.400 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:28:56.404 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:28:56.405 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-08-16T02:28:56.405 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv01.front.sepia.ceph.com 3 6 377 9 +91us[ +98us] +/- 45ms 2024-08-16T02:28:56.405 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -688us[ -688us] +/- 65ms 2024-08-16T02:28:56.405 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +592us[ +592us] +/- 65ms 2024-08-16T02:28:56.405 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:28:56.406 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-16T02:28:56.505 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-16T02:28:56.505 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-16T02:28:56.539 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-16T02:28:56.600 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:28:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:28:56.627 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-16T02:28:56.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-16T02:28:56.670 INFO:teuthology.task.internal:Duration was 1076.876709 seconds 2024-08-16T02:28:56.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-16T02:28:56.705 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-16T02:28:56.706 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-16T02:28:56.744 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:28:56.902 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-16T02:28:56.903 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-08-16T02:28:56.903 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-16T02:28:56.963 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-16T02:28:56.964 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-16T02:28:57.030 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-16T02:28:57.030 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:28:57.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-16T02:28:57.179 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-16T02:28:57.179 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-16T02:28:57.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-16T02:28:57.214 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:28:57.268 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-08-16T02:28:57.298 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:28:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:28:57.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-16T02:28:57.366 INFO:teuthology.task.internal:Transferring archived files... 2024-08-16T02:28:57.367 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857403/remote/smithi087 2024-08-16T02:28:57.367 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-16T02:28:57.425 INFO:teuthology.task.internal:Removing archive directory... 2024-08-16T02:28:57.426 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-16T02:28:57.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-16T02:28:57.487 INFO:teuthology.task.internal:Not uploading archives. 2024-08-16T02:28:57.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-16T02:28:57.515 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-16T02:28:57.516 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-16T02:28:57.530 INFO:teuthology.orchestra.run.smithi087.stdout: 265497 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:28 /home/ubuntu/cephtest 2024-08-16T02:28:57.530 INFO:teuthology.orchestra.run.smithi087.stdout: 406091 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:19 /home/ubuntu/cephtest/mnt.0 2024-08-16T02:28:57.531 INFO:teuthology.orchestra.run.smithi087.stdout: 406092 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:20 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-16T02:28:57.531 INFO:teuthology.orchestra.run.smithi087.stdout: 420903 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:20 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-16T02:28:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:28:57.532 INFO:teuthology.orchestra.run.smithi087.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-16T02:28:57.532 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-16T02:28:57.538 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-16T02:28:57.549 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-16T02:28:57.589 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi087 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-16T02:28:57.596 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format} duration: 1076.8767092227936 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi087 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=22f61a978a2346cc95dfc3d47864a60c status: fail success: false 2024-08-16T02:28:57.596 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-16T02:28:57.689 INFO:teuthology.run:FAIL