2024-08-16T02:07:58.687 INFO:root:teuthology version: 1.2.0 2024-08-16T02:07:58.702 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-16T02:07:58.784 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857404 branch: wip-yingxin-testing-2024-08-15-1551 description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7857404' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yingxin-testing-2024-08-15-1551 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(POOL_FULL\) sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 workunit: branch: wip-yingxin-testing-2024-08-15-1551 env: CRIMSON_COMPAT: '1' sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-yingxin-testing-2024-08-15-1551 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/Xy0oxVNLNkkiQ3oMX7uAUclROjFPMFS4grXfF5U1uWymw4Y35uaqbzOy7sh9PoqAAn7VI3lgsU34YiTZecj8= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-16_01:59:29 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-16T02:07:58.784 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa; will attempt to use it 2024-08-16T02:07:58.785 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks 2024-08-16T02:07:58.785 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-16T02:07:58.786 INFO:teuthology.task.internal:Checking packages... 2024-08-16T02:07:58.807 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '6f386a3f0b1fe6a4d7219b54bf963d790bc84004' 2024-08-16T02:07:58.807 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-16T02:07:58.807 INFO:teuthology.packaging:ref: None 2024-08-16T02:07:58.807 INFO:teuthology.packaging:tag: None 2024-08-16T02:07:58.807 INFO:teuthology.packaging:branch: wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:07:58.807 INFO:teuthology.packaging:sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:07:58.807 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:07:58.957 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5812.g6f386a3f 2024-08-16T02:07:58.959 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-16T02:07:58.965 INFO:teuthology.task.internal:no buildpackages task found 2024-08-16T02:07:58.965 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-16T02:07:58.971 INFO:teuthology.task.internal:Saving configuration 2024-08-16T02:07:58.982 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-16T02:07:58.988 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-16T02:07:59.013 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857404', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-16 02:03:00.634294', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/Xy0oxVNLNkkiQ3oMX7uAUclROjFPMFS4grXfF5U1uWymw4Y35uaqbzOy7sh9PoqAAn7VI3lgsU34YiTZecj8='} 2024-08-16T02:07:59.013 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-16T02:07:59.019 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-08-16T02:07:59.019 INFO:teuthology.run_tasks:Running task console_log... 2024-08-16T02:07:59.049 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbc75243ac0>, signals=[15]) 2024-08-16T02:07:59.049 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-16T02:07:59.055 INFO:teuthology.task.internal:Opening connections... 2024-08-16T02:07:59.055 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:07:59.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:07:59.135 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-16T02:07:59.148 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-08-16T02:07:59.220 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-08-16T02:07:59.221 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-08-16T02:07:59.278 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-08-16T02:07:59.278 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-08-16T02:07:59.278 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-08-16T02:07:59.278 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-08-16T02:07:59.278 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-08-16T02:07:59.278 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-08-16T02:07:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-16T02:07:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-08-16T02:07:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-08-16T02:07:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-16T02:07:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-08-16T02:07:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-16T02:07:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-16T02:07:59.279 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-16T02:07:59.280 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-08-16T02:07:59.297 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-16T02:07:59.305 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-16T02:07:59.317 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-16T02:07:59.317 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-08-16T02:07:59.335 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-16T02:07:59.340 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-16T02:07:59.340 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-08-16T02:07:59.402 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-16T02:07:59.436 INFO:teuthology.run_tasks:Running task kernel... 2024-08-16T02:07:59.450 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-08-16T02:07:59.450 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-08-16T02:07:59.451 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-16T02:07:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:07:59.467 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-08-16T02:07:59.525 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-437.el9.x86_64 2024-08-16T02:07:59.525 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-437.el9.x86_64 2024-08-16T02:07:59.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-08-16T02:08:01.623 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-08-16T02:08:10.747 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 20 MB 00:07 2024-08-16T02:08:15.206 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.6 MB 00:00 2024-08-16T02:08:16.715 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 18 kB 00:00 2024-08-16T02:08:17.752 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 25 MB/s | 22 MB 00:00 2024-08-16T02:08:23.442 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-08-16T02:08:25.481 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-16T02:08:25.482 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-16T02:08:25.576 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:08:25.586 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:08:25.587 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:08:25.587 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-08-16T02:08:25.587 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:08:25.587 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 84 M 2024-08-16T02:08:25.587 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 126 M 2024-08-16T02:08:25.587 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:08:26.151 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 5.3 MB/s | 1.5 MB 00:00 2024-08-16T02:08:26.343 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 36 MB/s | 17 MB 00:00 2024-08-16T02:08:26.710 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 54 MB/s | 30 MB 00:00 2024-08-16T02:08:27.219 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 27 MB/s | 36 MB 00:01 2024-08-16T02:08:27.222 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:08:27.222 INFO:teuthology.orchestra.run.smithi067.stdout:Total 52 MB/s | 84 MB 00:01 2024-08-16T02:08:27.989 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:08:28.399 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:08:28.400 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:08:29.439 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:08:29.442 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:08:32.023 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:08:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:08:32.811 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:08:35.599 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:08:35.746 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:08:40.957 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:08:41.014 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:08:46.242 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:17.594 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:17.636 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:18.010 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:18.010 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:09:18.010 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:09:18.010 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:09:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:09:18.842 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:09:18.843 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:09:18.843 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-16T02:09:18.843 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-16T02:09:18.843 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-16T02:09:18.843 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-16T02:09:18.843 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:09:18.843 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:09:19.194 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:09:19.277 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:09:19.277 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:09:19.277 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-16T02:09:19.277 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-16T02:09:19.278 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-16T02:09:19.278 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-16T02:09:19.278 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-08-16T02:09:19.278 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:09:19.336 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:09:19.336 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:09:19.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-08-16T02:09:19.374 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-16T02:09:19.375 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:09:19.376 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-08-16T02:09:19.377 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-16T02:09:19.377 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.14.0-480.el9.x86_64 2024-08-16T02:09:19.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-16T02:09:19.995 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-08-16T02:09:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-16T02:09:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-08-16T02:09:21.361 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-08-16T02:09:21.377 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.EYKiFDCPkI 2024-08-16T02:09:21.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EYKiFDCPkI 2024-08-16T02:09:21.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.EYKiFDCPkI 2024-08-16T02:09:21.580 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.EYKiFDCPkI is 6KB 2024-08-16T02:09:21.630 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.EYKiFDCPkI 2024-08-16T02:09:21.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-08-16T02:09:21.707 INFO:teuthology.orchestra.run.smithi067.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-16T02:09:21.708 INFO:teuthology.orchestra.run.smithi067.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-16T02:09:21.708 INFO:teuthology.orchestra.run.smithi067.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-16T02:09:21.708 INFO:teuthology.orchestra.run.smithi067.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-16T02:09:21.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-16T02:09:21.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-08-16T02:09:51.886 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-16T02:09:51.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-08-16T02:09:51.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:10:26.399 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-08-16T02:10:35.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-08-16T02:10:35.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:10:53.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-08-16T02:11:05.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-08-16T02:11:05.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:11:07.018 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-08-16T02:11:07.472 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-08-16T02:11:07.472 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-16T02:11:07.473 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-08-16T02:11:07.488 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-480.el9.x86_64 2024-08-16T02:11:07.488 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-16T02:11:07.488 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-16T02:11:07.488 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-16T02:11:08.490 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-16T02:11:08.490 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-16T02:11:08.562 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-08-16T02:11:08.601 DEBUG:teuthology.parallel:result is None 2024-08-16T02:11:08.601 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-16T02:11:08.607 INFO:teuthology.task.internal:Creating test directory... 2024-08-16T02:11:08.608 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-16T02:11:08.623 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-16T02:11:08.629 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-16T02:11:08.636 INFO:teuthology.task.internal:Creating archive directory... 2024-08-16T02:11:08.636 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-16T02:11:08.695 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-16T02:11:08.703 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-16T02:11:08.703 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-16T02:11:08.762 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:11:08.772 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:11:08.774 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-16T02:11:08.780 INFO:teuthology.task.internal:Configuring sudo... 2024-08-16T02:11:08.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-16T02:11:08.839 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-16T02:11:08.852 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-16T02:11:08.852 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:11:08.895 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:11:08.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:11:09.030 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:11:09.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:11:09.155 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:11:09.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-16T02:11:09.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-08-16T02:11:09.316 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:11:09.390 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-16T02:11:09.399 INFO:teuthology.task.internal:Starting timer... 2024-08-16T02:11:09.399 INFO:teuthology.run_tasks:Running task pcp... 2024-08-16T02:11:09.412 INFO:teuthology.run_tasks:Running task selinux... 2024-08-16T02:11:09.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-08-16T02:11:09.475 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-08-16T02:11:09.477 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-16T02:11:09.477 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-08-16T02:11:09.501 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-08-16T02:11:09.502 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive'} 2024-08-16T02:11:09.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:11:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:11:09.568 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-16T02:11:09.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-08-16T02:11:09.634 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-16T02:11:09.643 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-16T02:11:09.655 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-16T02:11:09.655 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-16T02:15:51.125 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com')] 2024-08-16T02:15:51.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-08-16T02:15:51.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:15:51.207 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-08-16T02:15:51.290 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-08-16T02:15:51.291 INFO:teuthology.run_tasks:Running task clock... 2024-08-16T02:15:51.301 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-16T02:15:51.301 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-16T02:15:51.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:15:51.361 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-16T02:15:51.375 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-16T02:15:51.400 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-08-16T02:15:51.412 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-08-16T02:15:51.426 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-16T02:15:51.441 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-16T02:15:51.505 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:15:51.508 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:15:51.508 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-08-16T02:15:51.509 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:15:51.509 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:15:51.509 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:15:51.509 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:15:51.510 INFO:teuthology.run_tasks:Running task install... 2024-08-16T02:15:51.518 DEBUG:teuthology.task.install:project ceph 2024-08-16T02:15:51.518 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}} 2024-08-16T02:15:51.519 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'} 2024-08-16T02:15:51.519 INFO:teuthology.task.install:Using flavor: crimson 2024-08-16T02:15:51.524 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-16T02:15:51.525 INFO:teuthology.task.install:extra packages: [] 2024-08-16T02:15:51.525 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'tag': None, 'wait_for_package': False} 2024-08-16T02:15:51.525 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:15:51.669 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/ 2024-08-16T02:15:51.670 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5812.g6f386a3f 2024-08-16T02:15:51.777 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-16T02:15:51.777 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:15:51.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-16T02:15:51.805 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-16T02:15:51.806 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-16T02:15:51.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-16T02:15:51.951 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-16T02:15:52.017 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-08-16T02:15:52.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-08-16T02:15:52.263 INFO:teuthology.orchestra.run.smithi067.stdout:45 files removed 2024-08-16T02:15:52.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-08-16T02:15:52.918 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 264 kB/s | 88 kB 00:00 2024-08-16T02:15:53.276 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 79 kB/s | 24 kB 00:00 2024-08-16T02:15:53.603 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 7.4 kB/s | 2.1 kB 00:00 2024-08-16T02:15:55.410 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-08-16T02:15:59.636 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 20 MB 00:03 2024-08-16T02:16:04.021 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.6 MB 00:00 2024-08-16T02:16:05.596 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 18 kB 00:00 2024-08-16T02:16:06.609 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 25 MB/s | 22 MB 00:00 2024-08-16T02:16:12.324 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-08-16T02:16:14.778 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:16:14.788 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 10 M 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.8 M 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.1 M 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 20 M 2024-08-16T02:16:14.789 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 25 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 807 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 558 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.4 M 2024-08-16T02:16:14.790 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 45 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 138 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 185 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 364 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 357 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 117 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-16T02:16:14.791 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-16T02:16:14.792 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-16T02:16:14.792 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-08-16T02:16:14.792 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-16T02:16:14.792 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:14.792 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:16:14.792 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:16:14.793 INFO:teuthology.orchestra.run.smithi067.stdout:Install 30 Packages 2024-08-16T02:16:14.793 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-08-16T02:16:14.793 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:14.793 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 60 M 2024-08-16T02:16:14.793 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:16:15.833 INFO:teuthology.orchestra.run.smithi067.stdout:(1/32): ceph-base-19.0.0-5812.g6f386a3f.el9.x86 11 MB/s | 6.1 MB 00:00 2024-08-16T02:16:15.875 INFO:teuthology.orchestra.run.smithi067.stdout:(2/32): ceph-selinux-19.0.0-5812.g6f386a3f.el9. 604 kB/s | 25 kB 00:00 2024-08-16T02:16:15.975 INFO:teuthology.orchestra.run.smithi067.stdout:(3/32): libcephfs2-19.0.0-5812.g6f386a3f.el9.x8 7.9 MB/s | 807 kB 00:00 2024-08-16T02:16:16.067 INFO:teuthology.orchestra.run.smithi067.stdout:(4/32): libradosstriper1-19.0.0-5812.g6f386a3f. 6.0 MB/s | 558 kB 00:00 2024-08-16T02:16:16.384 INFO:teuthology.orchestra.run.smithi067.stdout:(5/32): ceph-common-19.0.0-5812.g6f386a3f.el9.x 17 MB/s | 20 MB 00:01 2024-08-16T02:16:16.501 INFO:teuthology.orchestra.run.smithi067.stdout:(6/32): ceph-radosgw-19.0.0-5812.g6f386a3f.el9. 8.3 MB/s | 10 MB 00:01 2024-08-16T02:16:16.518 INFO:teuthology.orchestra.run.smithi067.stdout:(7/32): python3-ceph-argparse-19.0.0-5812.g6f38 338 kB/s | 45 kB 00:00 2024-08-16T02:16:16.551 INFO:teuthology.orchestra.run.smithi067.stdout:(8/32): python3-ceph-common-19.0.0-5812.g6f386a 2.7 MB/s | 138 kB 00:00 2024-08-16T02:16:16.577 INFO:teuthology.orchestra.run.smithi067.stdout:(9/32): python3-cephfs-19.0.0-5812.g6f386a3f.el 3.1 MB/s | 185 kB 00:00 2024-08-16T02:16:16.602 INFO:teuthology.orchestra.run.smithi067.stdout:(10/32): python3-rados-19.0.0-5812.g6f386a3f.el 7.1 MB/s | 364 kB 00:00 2024-08-16T02:16:16.627 INFO:teuthology.orchestra.run.smithi067.stdout:(11/32): python3-rbd-19.0.0-5812.g6f386a3f.el9. 6.9 MB/s | 357 kB 00:00 2024-08-16T02:16:16.653 INFO:teuthology.orchestra.run.smithi067.stdout:(12/32): python3-rgw-19.0.0-5812.g6f386a3f.el9. 2.3 MB/s | 117 kB 00:00 2024-08-16T02:16:16.736 INFO:teuthology.orchestra.run.smithi067.stdout:(13/32): librgw2-19.0.0-5812.g6f386a3f.el9.x86_ 8.1 MB/s | 5.4 MB 00:00 2024-08-16T02:16:16.887 INFO:teuthology.orchestra.run.smithi067.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 413 kB/s | 107 kB 00:00 2024-08-16T02:16:16.904 INFO:teuthology.orchestra.run.smithi067.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 271 kB/s | 45 kB 00:00 2024-08-16T02:16:16.962 INFO:teuthology.orchestra.run.smithi067.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 518 kB/s | 160 kB 00:00 2024-08-16T02:16:17.012 INFO:teuthology.orchestra.run.smithi067.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-08-16T02:16:17.054 INFO:teuthology.orchestra.run.smithi067.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-08-16T02:16:17.096 INFO:teuthology.orchestra.run.smithi067.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-08-16T02:16:17.113 INFO:teuthology.orchestra.run.smithi067.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 615 kB/s | 36 kB 00:00 2024-08-16T02:16:17.139 INFO:teuthology.orchestra.run.smithi067.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-08-16T02:16:17.164 INFO:teuthology.orchestra.run.smithi067.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 702 kB/s | 47 kB 00:00 2024-08-16T02:16:17.281 INFO:teuthology.orchestra.run.smithi067.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-08-16T02:16:17.306 INFO:teuthology.orchestra.run.smithi067.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-16T02:16:17.331 INFO:teuthology.orchestra.run.smithi067.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 227 kB/s | 26 kB 00:00 2024-08-16T02:16:17.357 INFO:teuthology.orchestra.run.smithi067.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-16T02:16:17.390 INFO:teuthology.orchestra.run.smithi067.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.7 MB/s | 192 kB 00:00 2024-08-16T02:16:17.432 INFO:teuthology.orchestra.run.smithi067.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-08-16T02:16:17.499 INFO:teuthology.orchestra.run.smithi067.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-08-16T02:16:17.599 INFO:teuthology.orchestra.run.smithi067.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-08-16T02:16:17.700 INFO:teuthology.orchestra.run.smithi067.stdout:(31/32): librados2-19.0.0-5812.g6f386a3f.el9.x8 13 MB/s | 3.5 MB 00:00 2024-08-16T02:16:17.791 INFO:teuthology.orchestra.run.smithi067.stdout:(32/32): librbd1-19.0.0-5812.g6f386a3f.el9.x86_ 13 MB/s | 3.8 MB 00:00 2024-08-16T02:16:17.796 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:16:17.796 INFO:teuthology.orchestra.run.smithi067.stdout:Total 20 MB/s | 60 MB 00:03 2024-08-16T02:16:17.899 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:16:17.940 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:16:17.940 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:16:18.541 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:16:18.541 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:16:19.971 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:16:20.511 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-16T02:16:20.577 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:16:20.666 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:16:20.762 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:16:20.787 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:16:20.850 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:16:20.912 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-08-16T02:16:20.942 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-08-16T02:16:20.967 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-08-16T02:16:20.990 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:16:21.014 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:16:21.210 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:16:21.240 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:16:21.621 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-08-16T02:16:21.669 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:16:21.720 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:16:21.765 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 12/34 2024-08-16T02:16:21.823 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:16:21.850 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 14/34 2024-08-16T02:16:21.932 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-08-16T02:16:21.955 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:16:22.078 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:16:22.112 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-08-16T02:16:22.223 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-08-16T02:16:22.263 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 19/34 2024-08-16T02:16:22.305 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-08-16T02:16:22.337 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-08-16T02:16:22.366 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 22/34 2024-08-16T02:16:22.397 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-08-16T02:16:22.433 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-16T02:16:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-16T02:16:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:16:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:16:23.213 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 27/34 2024-08-16T02:16:23.239 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 28/34 2024-08-16T02:16:25.850 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:16:25.894 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:16:26.635 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:16:26.673 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:16:26.752 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:16:26.753 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-16T02:16:26.753 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-16T02:16:26.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:26.782 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:16:37.399 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:16:37.400 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-08-16T02:16:37.400 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-08-16T02:16:37.400 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-08-16T02:16:37.400 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-08-16T02:16:37.400 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-08-16T02:16:37.400 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-08-16T02:16:37.400 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-08-16T02:16:37.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:38.529 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:16:38.556 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:16:38.556 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:16:38.556 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-16T02:16:38.556 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:16:38.556 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:16:38.556 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:38.558 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:16:38.648 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:16:38.648 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/34 2024-08-16T02:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:16:39.867 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 6/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 7/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 8/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 9/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 10/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 12/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-16T02:16:39.868 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-16T02:16:39.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-16T02:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-16T02:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-16T02:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-16T02:16:39.870 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-16T02:16:39.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/34 2024-08-16T02:16:40.699 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.700 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.701 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:40.702 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:16:41.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-08-16T02:16:41.658 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:29 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:16:41.791 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 86 M 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout: libasan8 x86_64 13.3.1-2.1.el9 appstream 469 k 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 175 k 2024-08-16T02:16:41.792 INFO:teuthology.orchestra.run.smithi067.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout:Install 9 Packages 2024-08-16T02:16:41.793 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:16:41.794 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 88 M 2024-08-16T02:16:41.794 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 834 M 2024-08-16T02:16:41.794 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:16:42.535 INFO:teuthology.orchestra.run.smithi067.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 615 kB/s | 94 kB 00:00 2024-08-16T02:16:42.635 INFO:teuthology.orchestra.run.smithi067.stdout:(2/9): libcephsqlite-19.0.0-5812.g6f386a3f.el9. 690 kB/s | 175 kB 00:00 2024-08-16T02:16:43.144 INFO:teuthology.orchestra.run.smithi067.stdout:(3/9): libubsan-11.4.1-3.el9.x86_64.rpm 368 kB/s | 187 kB 00:00 2024-08-16T02:16:43.228 INFO:teuthology.orchestra.run.smithi067.stdout:(4/9): libasan8-13.3.1-2.1.el9.x86_64.rpm 678 kB/s | 469 kB 00:00 2024-08-16T02:16:43.344 INFO:teuthology.orchestra.run.smithi067.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-08-16T02:16:43.445 INFO:teuthology.orchestra.run.smithi067.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 637 kB/s | 64 kB 00:00 2024-08-16T02:16:43.506 INFO:teuthology.orchestra.run.smithi067.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-08-16T02:16:43.578 INFO:teuthology.orchestra.run.smithi067.stdout:(8/9): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-08-16T02:16:44.871 INFO:teuthology.orchestra.run.smithi067.stdout:(9/9): ceph-test-19.0.0-5812.g6f386a3f.el9.x86_ 35 MB/s | 86 MB 00:02 2024-08-16T02:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:16:44.874 INFO:teuthology.orchestra.run.smithi067.stdout:Total 29 MB/s | 88 MB 00:03 2024-08-16T02:16:44.959 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:16:44.985 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:16:44.985 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:16:45.624 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:16:45.624 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:16:46.577 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:16:46.636 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-08-16T02:16:46.690 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-08-16T02:16:46.734 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-08-16T02:16:46.760 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-08-16T02:16:46.799 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:16:46.824 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:16:46.885 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:16:46.935 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-08-16T02:16:46.959 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:16:59.249 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:16:59.328 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:17:05.545 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:17:05.545 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/9 2024-08-16T02:17:05.545 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/9 2024-08-16T02:17:05.545 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-08-16T02:17:05.545 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libasan8-13.3.1-2.1.el9.x86_64 4/9 2024-08-16T02:17:05.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:17:05.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-08-16T02:17:05.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-08-16T02:17:05.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-08-16T02:17:05.902 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-08-16T02:17:05.902 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:05.902 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:17:05.902 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:05.902 INFO:teuthology.orchestra.run.smithi067.stdout: libasan8-13.3.1-2.1.el9.x86_64 2024-08-16T02:17:05.902 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:05.902 INFO:teuthology.orchestra.run.smithi067.stdout: libubsan-11.4.1-3.el9.x86_64 2024-08-16T02:17:05.903 INFO:teuthology.orchestra.run.smithi067.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-08-16T02:17:05.903 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-16T02:17:05.903 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-16T02:17:05.903 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-16T02:17:05.903 INFO:teuthology.orchestra.run.smithi067.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-08-16T02:17:05.903 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:05.903 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:17:06.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-08-16T02:17:06.767 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:54 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:17:06.904 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:17:06.906 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:06.906 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:17:06.906 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:06.906 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.5 k 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 2.2 M 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 1.6 M 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 284 k 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.1 M 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 72 M 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-16T02:17:06.907 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-16T02:17:06.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-16T02:17:06.909 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout:Install 37 Packages 2024-08-16T02:17:06.910 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:06.911 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 86 M 2024-08-16T02:17:06.911 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 943 M 2024-08-16T02:17:06.911 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:17:07.793 INFO:teuthology.orchestra.run.smithi067.stdout:(1/37): ceph-19.0.0-5812.g6f386a3f.el9.x86_64.r 20 kB/s | 6.5 kB 00:00 2024-08-16T02:17:08.344 INFO:teuthology.orchestra.run.smithi067.stdout:(2/37): ceph-mgr-19.0.0-5812.g6f386a3f.el9.x86_ 1.8 MB/s | 1.6 MB 00:00 2024-08-16T02:17:08.711 INFO:teuthology.orchestra.run.smithi067.stdout:(3/37): ceph-mds-19.0.0-5812.g6f386a3f.el9.x86_ 1.8 MB/s | 2.2 MB 00:01 2024-08-16T02:17:08.827 INFO:teuthology.orchestra.run.smithi067.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5812.g6f38 2.4 MB/s | 284 kB 00:00 2024-08-16T02:17:08.936 INFO:teuthology.orchestra.run.smithi067.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-08-16T02:17:09.028 INFO:teuthology.orchestra.run.smithi067.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 13 MB/s | 1.2 MB 00:00 2024-08-16T02:17:09.095 INFO:teuthology.orchestra.run.smithi067.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-08-16T02:17:09.153 INFO:teuthology.orchestra.run.smithi067.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-08-16T02:17:09.237 INFO:teuthology.orchestra.run.smithi067.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.5 MB/s | 125 kB 00:00 2024-08-16T02:17:09.320 INFO:teuthology.orchestra.run.smithi067.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-08-16T02:17:09.521 INFO:teuthology.orchestra.run.smithi067.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 942 kB/s | 188 kB 00:00 2024-08-16T02:17:09.604 INFO:teuthology.orchestra.run.smithi067.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-08-16T02:17:09.655 INFO:teuthology.orchestra.run.smithi067.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 693 kB/s | 35 kB 00:00 2024-08-16T02:17:09.697 INFO:teuthology.orchestra.run.smithi067.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-08-16T02:17:09.780 INFO:teuthology.orchestra.run.smithi067.stdout:(15/37): ceph-mon-19.0.0-5812.g6f386a3f.el9.x86 2.5 MB/s | 5.1 MB 00:01 2024-08-16T02:17:09.805 INFO:teuthology.orchestra.run.smithi067.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-08-16T02:17:09.923 INFO:teuthology.orchestra.run.smithi067.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 294 kB/s | 42 kB 00:00 2024-08-16T02:17:09.989 INFO:teuthology.orchestra.run.smithi067.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-08-16T02:17:10.015 INFO:teuthology.orchestra.run.smithi067.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-08-16T02:17:10.040 INFO:teuthology.orchestra.run.smithi067.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-08-16T02:17:10.099 INFO:teuthology.orchestra.run.smithi067.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-08-16T02:17:10.116 INFO:teuthology.orchestra.run.smithi067.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 636 kB/s | 11 kB 00:00 2024-08-16T02:17:10.132 INFO:teuthology.orchestra.run.smithi067.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-08-16T02:17:10.150 INFO:teuthology.orchestra.run.smithi067.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-08-16T02:17:10.166 INFO:teuthology.orchestra.run.smithi067.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 1.2 MB/s | 19 kB 00:00 2024-08-16T02:17:10.183 INFO:teuthology.orchestra.run.smithi067.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 1.1 MB/s | 20 kB 00:00 2024-08-16T02:17:10.200 INFO:teuthology.orchestra.run.smithi067.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 2.7 MB/s | 46 kB 00:00 2024-08-16T02:17:10.217 INFO:teuthology.orchestra.run.smithi067.stdout:(28/37): python3-more-itertools-8.12.0-2.el9.no 4.6 MB/s | 79 kB 00:00 2024-08-16T02:17:10.251 INFO:teuthology.orchestra.run.smithi067.stdout:(29/37): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-08-16T02:17:10.276 INFO:teuthology.orchestra.run.smithi067.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 658 kB/s | 16 kB 00:00 2024-08-16T02:17:10.310 INFO:teuthology.orchestra.run.smithi067.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-08-16T02:17:10.377 INFO:teuthology.orchestra.run.smithi067.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 537 kB/s | 36 kB 00:00 2024-08-16T02:17:10.402 INFO:teuthology.orchestra.run.smithi067.stdout:(33/37): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-08-16T02:17:10.452 INFO:teuthology.orchestra.run.smithi067.stdout:(34/37): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-08-16T02:17:10.486 INFO:teuthology.orchestra.run.smithi067.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 648 kB/s | 20 kB 00:00 2024-08-16T02:17:10.519 INFO:teuthology.orchestra.run.smithi067.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-08-16T02:17:14.203 INFO:teuthology.orchestra.run.smithi067.stdout:(37/37): ceph-osd-19.0.0-5812.g6f386a3f.el9.x86 12 MB/s | 72 MB 00:05 2024-08-16T02:17:14.208 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:14.208 INFO:teuthology.orchestra.run.smithi067.stdout:Total 12 MB/s | 86 MB 00:07 2024-08-16T02:17:14.363 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:17:14.410 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:17:14.410 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:17:15.221 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:17:15.221 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:17:16.936 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:17:16.998 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-16T02:17:17.039 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-16T02:17:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-16T02:17:17.149 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-16T02:17:17.208 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-16T02:17:17.241 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-16T02:17:17.336 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-16T02:17:17.414 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-16T02:17:17.460 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-16T02:17:17.495 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-16T02:17:17.525 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-16T02:17:17.559 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-16T02:17:17.592 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-16T02:17:17.634 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-16T02:17:17.677 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-16T02:17:17.711 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-16T02:17:17.760 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-16T02:17:17.801 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-16T02:17:17.878 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-16T02:17:18.039 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-16T02:17:18.128 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-16T02:17:18.187 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-16T02:17:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-16T02:17:18.284 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-16T02:17:18.348 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-16T02:17:18.556 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-16T02:17:18.612 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-16T02:17:18.694 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-16T02:17:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-16T02:17:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-16T02:17:18.964 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-16T02:17:19.166 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 32/37 2024-08-16T02:17:19.188 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:17:19.214 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:17:19.215 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:19.215 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-16T02:17:19.215 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:17:19.215 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:17:19.215 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:30.914 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:17:30.944 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:17:30.944 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:30.944 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-16T02:17:30.944 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:17:30.944 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:17:30.944 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:31.506 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:17:31.532 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:17:31.532 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:31.532 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-16T02:17:31.532 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:17:31.532 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:17:31.532 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:31.782 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:17:31.807 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:17:31.807 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:31.807 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-16T02:17:31.807 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:17:31.808 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:17:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:31.873 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:17:38.593 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 6/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-16T02:17:38.594 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-16T02:17:38.595 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-16T02:17:38.596 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-16T02:17:38.596 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-16T02:17:38.597 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-16T02:17:38.597 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-16T02:17:38.598 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-16T02:17:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-16T02:17:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:39.563 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:17:39.563 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-16T02:17:39.564 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-16T02:17:39.565 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-16T02:17:39.566 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-16T02:17:39.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:39.567 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:17:39.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-08-16T02:17:40.418 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:28 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:17:40.508 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:17:40.551 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:17:40.552 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:17:40.553 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:17:40.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-08-16T02:17:41.178 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:29 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:17:41.311 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 773 k 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 773 k 2024-08-16T02:17:41.313 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 779 k 2024-08-16T02:17:41.313 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:17:41.638 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-19.0.0-5812.g6f386a3f.el9.noarch.rpm 2.3 MB/s | 773 kB 00:00 2024-08-16T02:17:41.638 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:41.639 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.3 MB/s | 773 kB 00:00 2024-08-16T02:17:41.639 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:17:41.644 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:17:41.644 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:17:41.650 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:17:41.651 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:17:41.727 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:17:42.209 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:17:42.311 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:17:42.688 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:17:42.943 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:17:42.943 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:42.943 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:17:42.943 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:42.943 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:42.943 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:17:43.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-08-16T02:17:43.753 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:31 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:17:43.888 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:17:43.888 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 158 k 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 158 k 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 644 k 2024-08-16T02:17:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:17:44.126 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-19.0.0-5812.g6f386a 669 kB/s | 158 kB 00:00 2024-08-16T02:17:44.126 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:44.127 INFO:teuthology.orchestra.run.smithi067.stdout:Total 665 kB/s | 158 kB 00:00 2024-08-16T02:17:44.127 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:17:44.138 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:17:44.138 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:17:44.198 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:17:44.198 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:17:44.352 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:17:44.419 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:17:44.444 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:17:44.445 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:17:44.445 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-16T02:17:44.445 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:45.374 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:17:45.375 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:45.375 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:17:45.375 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:17:45.375 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:45.375 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:17:45.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-08-16T02:17:46.180 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:34 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:17:46.269 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:17:46.313 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:17:46.314 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:17:46.314 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:17:46.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-08-16T02:17:46.951 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:34 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:17:47.082 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:17:47.083 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:47.083 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:17:47.083 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:47.083 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 14 M 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 34 k 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 16 k 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-16T02:17:47.084 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-16T02:17:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-16T02:17:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-16T02:17:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:47.085 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:17:47.085 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:17:47.085 INFO:teuthology.orchestra.run.smithi067.stdout:Install 11 Packages 2024-08-16T02:17:47.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:47.086 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 18 M 2024-08-16T02:17:47.086 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 154 M 2024-08-16T02:17:47.086 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:17:47.763 INFO:teuthology.orchestra.run.smithi067.stdout:(1/11): ceph-grafana-dashboards-19.0.0-5812.g6f 180 kB/s | 34 kB 00:00 2024-08-16T02:17:47.788 INFO:teuthology.orchestra.run.smithi067.stdout:(2/11): ceph-prometheus-alerts-19.0.0-5812.g6f3 76 kB/s | 16 kB 00:00 2024-08-16T02:17:47.980 INFO:teuthology.orchestra.run.smithi067.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-08-16T02:17:48.047 INFO:teuthology.orchestra.run.smithi067.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-08-16T02:17:48.106 INFO:teuthology.orchestra.run.smithi067.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.3 MB/s | 548 kB 00:00 2024-08-16T02:17:48.139 INFO:teuthology.orchestra.run.smithi067.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 217 kB/s | 19 kB 00:00 2024-08-16T02:17:48.173 INFO:teuthology.orchestra.run.smithi067.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-08-16T02:17:48.198 INFO:teuthology.orchestra.run.smithi067.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-08-16T02:17:48.223 INFO:teuthology.orchestra.run.smithi067.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-08-16T02:17:48.273 INFO:teuthology.orchestra.run.smithi067.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-08-16T02:17:48.782 INFO:teuthology.orchestra.run.smithi067.stdout:(11/11): ceph-mgr-dashboard-19.0.0-5812.g6f386a 12 MB/s | 14 MB 00:01 2024-08-16T02:17:48.786 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:17:48.786 INFO:teuthology.orchestra.run.smithi067.stdout:Total 11 MB/s | 18 MB 00:01 2024-08-16T02:17:48.817 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:17:48.837 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:17:48.838 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:17:49.089 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:17:49.090 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:17:49.658 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:17:49.740 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-08-16T02:17:49.769 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-08-16T02:17:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-08-16T02:17:50.002 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-08-16T02:17:50.056 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-08-16T02:17:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-08-16T02:17:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-08-16T02:17:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:17:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 9/11 2024-08-16T02:17:53.640 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 10/11 2024-08-16T02:17:53.697 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:17:55.672 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:17:55.672 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 1/11 2024-08-16T02:17:55.672 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 2/11 2024-08-16T02:17:55.672 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 3/11 2024-08-16T02:17:55.672 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-08-16T02:17:55.673 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-08-16T02:17:55.673 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-08-16T02:17:55.673 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-08-16T02:17:55.673 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:17:55.673 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-08-16T02:17:55.673 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-08-16T02:17:55.977 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-08-16T02:17:55.977 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:55.977 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:17:55.977 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-16T02:17:55.977 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:55.977 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:55.977 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:17:55.977 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-16T02:17:55.978 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-16T02:17:55.978 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-16T02:17:55.978 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-16T02:17:55.978 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-16T02:17:55.978 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-16T02:17:55.978 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-16T02:17:55.978 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:55.978 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:17:56.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-16T02:17:56.989 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:44 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:17:57.122 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:17:57.123 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 7.4 M 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-16T02:17:57.124 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-08-16T02:17:57.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:17:57.126 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 38 M 2024-08-16T02:17:57.126 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 209 M 2024-08-16T02:17:57.126 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:17:57.713 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-08-16T02:17:57.763 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-08-16T02:17:57.846 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-08-16T02:17:57.897 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-08-16T02:17:58.005 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-58 15 MB/s | 7.4 MB 00:00 2024-08-16T02:17:58.030 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 264 kB/s | 35 kB 00:00 2024-08-16T02:17:58.114 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-08-16T02:17:58.581 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-08-16T02:17:58.664 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-08-16T02:17:59.140 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.0 MB/s | 5.1 MB 00:01 2024-08-16T02:17:59.323 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.6 MB/s | 4.7 MB 00:01 2024-08-16T02:18:01.248 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.3 MB/s | 16 MB 00:02 2024-08-16T02:18:01.250 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:01.250 INFO:teuthology.orchestra.run.smithi067.stdout:Total 9.2 MB/s | 38 MB 00:04 2024-08-16T02:18:01.615 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:01.643 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:01.643 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:02.107 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:02.107 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:02.887 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:02.972 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-16T02:18:03.014 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:18:03.119 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:18:03.165 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-16T02:18:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-16T02:18:03.380 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-16T02:18:03.459 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-16T02:18:04.151 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-16T02:18:04.289 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-16T02:18:05.629 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:18:06.697 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-16T02:18:06.728 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:18:09.250 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 1/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:18:09.251 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-16T02:18:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-16T02:18:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-16T02:18:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-16T02:18:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-16T02:18:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:09.654 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:10.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-08-16T02:18:10.635 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:58 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:10.770 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 49 k 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-16T02:18:10.772 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-08-16T02:18:10.773 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:10.774 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.2 M 2024-08-16T02:18:10.774 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-08-16T02:18:10.774 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:11.289 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): ceph-mgr-rook-19.0.0-5812.g6f386a3f.el9 247 kB/s | 49 kB 00:00 2024-08-16T02:18:11.348 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 74 kB/s | 19 kB 00:00 2024-08-16T02:18:11.365 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 93 kB/s | 26 kB 00:00 2024-08-16T02:18:11.690 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-08-16T02:18:11.791 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 655 kB/s | 279 kB 00:00 2024-08-16T02:18:11.816 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 428 kB/s | 54 kB 00:00 2024-08-16T02:18:11.841 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 403 kB/s | 222 kB 00:00 2024-08-16T02:18:11.941 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-08-16T02:18:11.958 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 99 kB/s | 14 kB 00:00 2024-08-16T02:18:11.975 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-08-16T02:18:12.001 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-google-auth-2.33.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-08-16T02:18:12.026 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-08-16T02:18:12.059 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.8 MB/s | 1.0 MB 00:00 2024-08-16T02:18:12.062 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:12.062 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-08-16T02:18:12.128 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:12.144 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:12.144 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:12.239 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:12.240 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:12.932 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-16T02:18:12.997 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-16T02:18:13.053 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-16T02:18:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-16T02:18:13.130 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-16T02:18:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-16T02:18:13.348 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-16T02:18:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-16T02:18:14.159 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-16T02:18:14.212 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-16T02:18:14.246 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-16T02:18:14.295 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-16T02:18:14.323 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:18:15.043 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:18:15.043 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 1/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-16T02:18:15.044 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-16T02:18:15.367 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-16T02:18:15.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:15.367 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:15.367 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:18:15.367 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-16T02:18:15.367 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:15.368 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:15.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-08-16T02:18:16.184 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:04 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:16.320 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:16.320 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 150 k 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:16.321 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-08-16T02:18:16.322 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 7.0 M 2024-08-16T02:18:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 33 M 2024-08-16T02:18:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:16.989 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): ceph-mgr-cephadm-19.0.0-5812.g6f386a3f.e 635 kB/s | 150 kB 00:00 2024-08-16T02:18:17.114 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.3 MB/s | 548 kB 00:00 2024-08-16T02:18:17.139 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.4 MB/s | 58 kB 00:00 2024-08-16T02:18:17.164 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-08-16T02:18:17.690 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 265 kB/s | 248 kB 00:00 2024-08-16T02:18:21.056 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.4 MB/s | 6.0 MB 00:04 2024-08-16T02:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:21.059 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.5 MB/s | 7.0 MB 00:04 2024-08-16T02:18:21.135 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:21.148 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:21.148 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:21.255 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:21.256 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:21.585 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:21.774 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-16T02:18:21.826 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-16T02:18:22.388 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-16T02:18:22.490 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-16T02:18:22.563 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-16T02:18:22.588 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:18:23.340 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:18:23.341 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/6 2024-08-16T02:18:23.341 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-16T02:18:23.341 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-16T02:18:23.341 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-16T02:18:23.341 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-16T02:18:23.613 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:23.614 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:23.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-08-16T02:18:24.435 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:12 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:24.573 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 883 k 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:24.574 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:24.575 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-08-16T02:18:24.575 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:24.575 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 963 k 2024-08-16T02:18:24.575 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 4.1 M 2024-08-16T02:18:24.575 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:24.835 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 407 kB/s | 80 kB 00:00 2024-08-16T02:18:24.927 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-fuse-19.0.0-5812.g6f386a3f.el9.x86_ 3.0 MB/s | 883 kB 00:00 2024-08-16T02:18:24.927 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:24.928 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.7 MB/s | 963 kB 00:00 2024-08-16T02:18:24.940 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:24.965 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:24.965 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:25.049 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:25.049 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:25.240 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:25.386 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-16T02:18:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:18:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:18:26.155 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/2 2024-08-16T02:18:26.430 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-16T02:18:26.430 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:26.430 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:26.430 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-16T02:18:26.430 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:26.430 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:26.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-volume 2024-08-16T02:18:27.256 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:15 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:27.389 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:27.389 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 285 k 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 285 k 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.3 M 2024-08-16T02:18:27.390 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:27.620 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-volume-19.0.0-5812.g6f386a3f.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-08-16T02:18:27.620 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:27.621 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-08-16T02:18:27.621 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:27.630 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:27.630 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:27.659 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:27.660 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:27.887 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:27.957 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:18:27.973 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:18:27.974 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:18:27.974 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-16T02:18:27.974 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:29.009 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:18:29.009 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:29.009 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:29.009 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:18:29.010 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:29.010 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:29.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-08-16T02:18:29.909 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:17 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:30.043 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 125 k 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:30.044 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-08-16T02:18:30.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:30.045 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 125 k 2024-08-16T02:18:30.045 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 445 k 2024-08-16T02:18:30.045 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:30.275 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-19.0.0-5812.g6f386a3f.el9.x86_64 542 kB/s | 125 kB 00:00 2024-08-16T02:18:30.276 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:30.276 INFO:teuthology.orchestra.run.smithi067.stdout:Total 539 kB/s | 125 kB 00:00 2024-08-16T02:18:30.276 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:30.282 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:30.283 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:30.349 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:30.350 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:30.513 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:30.618 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:31.007 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:31.255 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:31.256 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:31.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-08-16T02:18:32.112 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:20 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:32.202 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:32.247 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:32.248 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:18:32.248 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:32.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-08-16T02:18:32.882 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:20 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:33.018 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:33.018 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-08-16T02:18:33.018 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 32 k 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 32 k 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 142 k 2024-08-16T02:18:33.019 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:33.182 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-19.0.0-5812.g6f386a3f.el9.x86_6 195 kB/s | 32 kB 00:00 2024-08-16T02:18:33.183 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:33.183 INFO:teuthology.orchestra.run.smithi067.stdout:Total 194 kB/s | 32 kB 00:00 2024-08-16T02:18:33.183 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:33.186 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:33.186 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:33.196 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:33.196 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:33.282 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:33.399 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:33.764 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:33.988 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:33.989 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:34.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-08-16T02:18:34.749 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:22 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:34.842 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:34.888 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:34.889 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:18:34.889 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:34.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-08-16T02:18:35.540 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:23 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:35.632 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:35.678 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:35.679 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:18:35.680 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:35.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-08-16T02:18:36.319 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:24 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:36.406 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:36.453 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:36.454 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:18:36.454 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:36.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-08-16T02:18:37.101 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:25 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:37.235 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:37.236 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:18:37.237 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:37.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-08-16T02:18:37.873 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:25 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:37.961 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:38.006 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:38.008 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:18:38.008 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:38.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-08-16T02:18:38.655 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:26 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:38.744 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:18:38.789 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:38.790 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-08-16T02:18:38.790 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:38.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-08-16T02:18:39.437 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:27 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:39.576 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:39.577 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-08-16T02:18:39.577 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:39.577 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 91 k 2024-08-16T02:18:39.577 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:39.577 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:39.577 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:39.578 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-08-16T02:18:39.578 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:39.578 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 91 k 2024-08-16T02:18:39.578 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 321 k 2024-08-16T02:18:39.578 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:39.785 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 440 kB/s | 91 kB 00:00 2024-08-16T02:18:39.785 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:39.786 INFO:teuthology.orchestra.run.smithi067.stdout:Total 437 kB/s | 91 kB 00:00 2024-08-16T02:18:39.786 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:39.796 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:39.796 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:39.862 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:39.863 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:40.017 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:40.135 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:40.491 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:40.752 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:40.752 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:40.752 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:40.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-08-16T02:18:41.580 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:29 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:41.717 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:41.717 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.5 M 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 30 M 2024-08-16T02:18:41.718 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:42.074 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 9.8 MB/s | 3.5 MB 00:00 2024-08-16T02:18:42.075 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:42.075 INFO:teuthology.orchestra.run.smithi067.stdout:Total 9.8 MB/s | 3.5 MB 00:00 2024-08-16T02:18:42.075 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:42.087 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:42.087 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:42.176 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:42.176 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:42.793 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:42.850 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:42.876 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:42.876 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:18:42.876 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-16T02:18:42.876 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:18:42.877 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:18:42.877 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:44.084 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:44.084 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:44.084 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:44.084 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:44.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:44.085 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:44.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-08-16T02:18:44.924 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:32 ago on Fri 16 Aug 2024 02:16:12 AM UTC. 2024-08-16T02:18:45.064 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-08-16T02:18:45.064 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:45.064 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-08-16T02:18:45.064 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 177 k 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 177 k 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 685 k 2024-08-16T02:18:45.065 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-08-16T02:18:45.293 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 776 kB/s | 177 kB 00:00 2024-08-16T02:18:45.294 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:18:45.294 INFO:teuthology.orchestra.run.smithi067.stdout:Total 771 kB/s | 177 kB 00:00 2024-08-16T02:18:45.294 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-08-16T02:18:45.305 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-08-16T02:18:45.305 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-08-16T02:18:45.373 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-08-16T02:18:45.373 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-08-16T02:18:45.538 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-08-16T02:18:45.652 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:46.040 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:46.295 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:18:46.295 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:46.295 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-08-16T02:18:46.295 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:18:46.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:46.296 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-08-16T02:18:46.536 DEBUG:teuthology.parallel:result is None 2024-08-16T02:18:46.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:18:46.685 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-16T02:18:46.708 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-5812.g6f386a3f.el9 2024-08-16T02:18:46.708 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5812.g6f386a3f.el9 2024-08-16T02:18:46.709 INFO:teuthology.task.install:The correct ceph version 19.0.0-5812.g6f386a3f is installed. 2024-08-16T02:18:46.711 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-16T02:18:46.711 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:46.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-16T02:18:46.775 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-16T02:18:46.776 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:46.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-08-16T02:18:46.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-16T02:18:46.912 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-16T02:18:46.912 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:46.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-16T02:18:46.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-16T02:18:47.042 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-16T02:18:47.042 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:47.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-08-16T02:18:47.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-16T02:18:47.173 INFO:teuthology.run_tasks:Running task ceph... 2024-08-16T02:18:47.251 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-16T02:18:47.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-08-16T02:18:47.277 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-16T02:18:47.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-16T02:18:47.343 INFO:tasks.ceph:Creating extra log directories... 2024-08-16T02:18:47.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-16T02:18:47.416 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-16T02:18:47.416 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-16T02:18:47.417 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857404', 'branch': 'wip-yingxin-testing-2024-08-15-1551', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7857404', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-testing-2024-08-15-1551'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}}, 'workunit': {'branch': 'wip-yingxin-testing-2024-08-15-1551', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-yingxin-testing-2024-08-15-1551', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/Xy0oxVNLNkkiQ3oMX7uAUclROjFPMFS4grXfF5U1uWymw4Y35uaqbzOy7sh9PoqAAn7VI3lgsU34YiTZecj8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '67db276914d29505c9707d04f3a3b8fb9d8aee4b', 'timestamp': '2024-08-16_01:59:29', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-16T02:18:47.417 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-16T02:18:47.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-08-16T02:18:47.532 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:47.532 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-08-16T02:18:47.588 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-16T02:18:47.588 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-08-16T02:18:47.643 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-16T02:18:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:18:47.644 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-08-16T02:18:47.644 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:18:47.644 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:18:47.644 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-08-16 02:18:43.143008911 +0000 2024-08-16T02:18:47.644 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-08-16 02:15:08.151985683 +0000 2024-08-16T02:18:47.644 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-08-16 02:15:08.151985683 +0000 2024-08-16T02:18:47.644 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-08-16 02:15:08.151985683 +0000 2024-08-16T02:18:47.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-16T02:18:47.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-08-16T02:18:47.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-08-16T02:18:47.708 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000203368 s, 2.5 MB/s 2024-08-16T02:18:47.709 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-16T02:18:47.765 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-08-16T02:18:47.821 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-16T02:18:47.822 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:18:47.822 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-08-16T02:18:47.822 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:18:47.822 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:18:47.822 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-08-16 02:18:43.143008911 +0000 2024-08-16T02:18:47.822 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-08-16 02:15:08.505977740 +0000 2024-08-16T02:18:47.822 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-08-16 02:15:08.505977740 +0000 2024-08-16T02:18:47.822 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-08-16 02:15:08.505977740 +0000 2024-08-16T02:18:47.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-16T02:18:47.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-08-16T02:18:47.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-08-16T02:18:47.886 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000207085 s, 2.5 MB/s 2024-08-16T02:18:47.887 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-16T02:18:47.944 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-08-16 02:18:43.143008911 +0000 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-08-16 02:15:08.836970313 +0000 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-08-16 02:15:08.836970313 +0000 2024-08-16T02:18:48.001 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-08-16 02:15:08.836970313 +0000 2024-08-16T02:18:48.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-16T02:18:48.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-08-16T02:18:48.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-08-16T02:18:48.065 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000210118 s, 2.4 MB/s 2024-08-16T02:18:48.066 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-16T02:18:48.122 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-08-16T02:18:48.178 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-16T02:18:48.179 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:18:48.179 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-08-16T02:18:48.179 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:18:48.179 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:18:48.179 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-08-16 02:18:43.144008890 +0000 2024-08-16T02:18:48.179 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-08-16 02:15:09.190962370 +0000 2024-08-16T02:18:48.179 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-08-16 02:15:09.190962370 +0000 2024-08-16T02:18:48.179 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-08-16 02:15:09.190962370 +0000 2024-08-16T02:18:48.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-16T02:18:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-08-16T02:18:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-08-16T02:18:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000183978 s, 2.8 MB/s 2024-08-16T02:18:48.244 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-16T02:18:48.300 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-16T02:18:48.300 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-16T02:18:48.300 INFO:tasks.ceph:Generating config... 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-16T02:18:48.302 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-08-16T02:18:48.302 INFO:tasks.ceph:[osd] debug monc = 20 2024-08-16T02:18:48.302 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-08-16T02:18:48.302 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-08-16T02:18:48.302 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-16T02:18:48.303 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-16T02:18:48.303 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-08-16T02:18:48.303 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-08-16T02:18:48.303 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:18:48.303 INFO:tasks.ceph:[global] ms client mode = crc 2024-08-16T02:18:48.303 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-08-16T02:18:48.303 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-08-16T02:18:48.303 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-08-16T02:18:48.303 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-08-16T02:18:48.303 INFO:tasks.ceph:[global] ms service mode = crc 2024-08-16T02:18:48.303 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-16T02:18:48.303 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-16T02:18:48.303 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-08-16T02:18:48.304 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-08-16T02:18:48.304 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-08-16T02:18:48.304 INFO:tasks.ceph:Setting up mon.a... 2024-08-16T02:18:48.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-16T02:18:48.394 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-08-16T02:18:48.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-16T02:18:48.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:18:48.475 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67')] 2024-08-16T02:18:48.475 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-08-16T02:18:48.476 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:48.476 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-16T02:18:48.530 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --print /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:48.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:18:48.622+0000 7f5a86239f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:48.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:18:48.622+0000 7f5a86239f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:48.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:18:48.622+0000 7f5a86239f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:18:48.623 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:48.623 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid 734e1993-fc74-4287-90f2-01825483bae0 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 734e1993-fc74-4287-90f2-01825483bae0 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-08-16T02:18:48.623555+0000 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-08-16T02:18:48.623555+0000 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-08-16T02:18:48.624 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-08-16T02:18:48.625 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-16T02:18:48.641 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 734e1993-fc74-4287-90f2-01825483bae0... 2024-08-16T02:18:48.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-16T02:18:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-08-16T02:18:48.726 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-08-16T02:18:48.726 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-16T02:18:48.726 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-08-16T02:18:48.726 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.726 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-08-16T02:18:48.726 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-08-16T02:18:48.727 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-16T02:18:48.728 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-08-16T02:18:48.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.729 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-16T02:18:48.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.729 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-08-16T02:18:48.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.729 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = 172.21.15.67 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: ms client mode = crc 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: ms cluster mode = crc 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: ms mon client mode = crc 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: ms mon cluster mode = crc 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: ms mon service mode = crc 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: ms service mode = crc 2024-08-16T02:18:48.730 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 734e1993-fc74-4287-90f2-01825483bae0 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-08-16T02:18:48.731 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-08-16T02:18:48.732 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 20 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: crimson alien thread cpu cores = 6-7 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: crimson osd obc lru size = 10 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = seastore 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: osd shutdown pgref assert = True 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-08-16T02:18:48.733 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-08-16T02:18:48.734 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-16T02:18:48.735 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: mon min osdmap epochs = 50 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune interval = 2 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune min = 15 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune txsize = 2 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: paxos service trim min = 10 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default crimson = True 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:18:48.736 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout: crimson seastar cpu cores = 0-1 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout: crimson seastar cpu cores = 2-3 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-08-16T02:18:48.737 INFO:teuthology.orchestra.run.smithi067.stdout: crimson seastar cpu cores = 4-5 2024-08-16T02:18:48.738 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-16T02:18:48.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-16T02:18:48.845 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-16T02:18:48.845 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:48.845 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-16T02:18:48.861 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:48.861 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-16T02:18:48.917 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:18:48.918 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:48.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-16T02:18:48.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:18:48.993 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:18:48.993 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:49.049 INFO:tasks.ceph:Setting up mon nodes... 2024-08-16T02:18:49.050 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-16T02:18:49.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-16T02:18:49.150 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-16T02:18:49.153 INFO:tasks.ceph:Setting up mds nodes... 2024-08-16T02:18:49.154 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-16T02:18:49.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-16T02:18:49.204 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-16T02:18:49.217 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-16T02:18:49.217 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-16T02:18:49.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-16T02:18:49.242 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-16T02:18:49.243 INFO:tasks.ceph:role: osd.0 2024-08-16T02:18:49.243 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:18:49.243 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-16T02:18:49.315 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:18:49.316 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:18:49.316 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:18:49.316 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:18:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:18:49.316 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:18:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:18:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:18:49.317 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:18:49.317 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:18:49.830 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-08-16T02:18:49.833 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-08-16T02:18:49.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-16T02:18:49.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-16T02:18:50.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-16T02:18:50.068 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-16T02:18:50.068 INFO:tasks.ceph:role: osd.1 2024-08-16T02:18:50.068 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:18:50.068 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-16T02:18:50.142 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:18:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:18:50.142 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:18:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:18:50.143 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:18:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:18:50.143 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:18:50.143 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:18:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:18:50.143 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:18:50.595 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-08-16T02:18:50.597 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-08-16T02:18:50.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-16T02:18:50.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-16T02:18:50.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-16T02:18:50.818 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-08-16T02:18:50.818 INFO:tasks.ceph:role: osd.2 2024-08-16T02:18:50.818 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:18:50.818 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-16T02:18:50.891 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:18:50.892 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:18:51.417 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-08-16T02:18:51.419 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-08-16T02:18:51.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-08-16T02:18:51.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-16T02:18:51.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:51,945 seastar - Reactor backend: linux-aio 2024-08-16T02:18:51.955 INFO:teuthology.orchestra.run.smithi067.stderr:==29593==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:52.060 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x5f802a8 0x5f804ee 0x5f8071f 0x5f809f5 0x5fcf0f3 0x5fcf2f9 0x5f4f446 0x5f4f97a 0x5d929a1 0x5daabe3 0x70ee632 0x70ffe05 0x71012a2 0x7101327 0x7101429 0x7101521 0x710172e 0x7101914 0x7101c63 0x7102007 0x7102237 0x71024a1 0x710276e 0x710298c 0x7102a44 0x7102b0a 0x7102beb 0x5bfe1b8 0x5bfe2c2 0x5bfe37e 0x5bfe43a 0x5bfe4f6 0x5bfe643 0x5bfe7ce 0x5bfeb8f 0x5c1aad0 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x38e8d78 0x38e98d1 0x3788ab2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:52.061 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:52,074 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:18:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:52,074 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-08-16T02:18:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:DEBUG 2024-08-16 02:18:52,075 [shard 0:main] seastar - reactor::drain 2024-08-16T02:18:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:Suppressions used: 2024-08-16T02:18:52.648 INFO:teuthology.orchestra.run.smithi067.stderr: count bytes template 2024-08-16T02:18:52.648 INFO:teuthology.orchestra.run.smithi067.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:52.648 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-08-16T02:18:52.717 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:52,722 seastar - Reactor backend: linux-aio 2024-08-16T02:18:52.729 INFO:teuthology.orchestra.run.smithi067.stderr:==29591==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:52.794 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x4a068 0x35b7c 0x36eee 0x5f738e7 0x5f73c58 0x5f7403a 0x5fcf264 0x5f4f446 0x5f4f620 0x5f4fbac 0x5d929a1 0x5daabe3 0x70ee632 0x70ffe05 0x71012a2 0x7101327 0x7101429 0x7101521 0x710172e 0x7101914 0x7101c63 0x7102007 0x7102237 0x71024a1 0x710276e 0x710298c 0x7102a44 0x7102b0a 0x7102beb 0x5bfe1b8 0x5bfe2c2 0x5bfe37e 0x5bfe43a 0x5bfe4f6 0x5bfe643 0x5bfe7ce 0x5bfeb8f 0x5c1aad0 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:52.794 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.013 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bc7 0x4a0bd 0x35b7c 0x3629e 0xbac61ab 0xbac75e2 0xbac785c 0xbac7a04 0xbac97bf 0xbac99bb 0xbb0dc39 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:53.013 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.234 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362e5 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:53.234 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x3627f 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bdf 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.360 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8a28 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:53.360 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.399 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:53.400 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.468 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7f5a 0x9e4b36e 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:53.468 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x15983f 0xc9c6c 0xca7a6 0x35920f0 0x3afc7c2 0x3afc9ba 0xbadb0c0 0xbb1555f 0xbb158ea 0xbab6dd6 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:53.809 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362fa 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:53.809 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:54.881 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:54.881 INFO:teuthology.orchestra.run.smithi067.stderr:Suppressions used: 2024-08-16T02:18:54.881 INFO:teuthology.orchestra.run.smithi067.stderr: count bytes template 2024-08-16T02:18:54.881 INFO:teuthology.orchestra.run.smithi067.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:54.881 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:54.881 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-08-16T02:18:54.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-16T02:18:54.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:55.256 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:55.258 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:55,258 seastar - Reactor backend: linux-aio 2024-08-16T02:18:55.263 INFO:teuthology.orchestra.run.smithi067.stderr:==29638==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x5f474cb 0x5f68ecc 0x5f69166 0x5f4fd8c 0x5d929a1 0x5daabe3 0x70ee632 0x70ffe05 0x71012a2 0x7101327 0x7101429 0x7101521 0x710172e 0x7101914 0x7101c63 0x7102007 0x7102237 0x71024a1 0x710276e 0x710298c 0x7102a44 0x7102b0a 0x7102beb 0x5bfe1b8 0x5bfe2c2 0x5bfe37e 0x5bfe43a 0x5bfe4f6 0x5bfe643 0x5bfe7ce 0x5bfeb8f 0x5c1aad0 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x38e8d78 0x38e98d1 0x3788ab2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:55.368 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:55,381 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:18:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:55,381 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-08-16T02:18:55.381 INFO:teuthology.orchestra.run.smithi067.stderr:DEBUG 2024-08-16 02:18:55,382 [shard 0:main] seastar - reactor::drain 2024-08-16T02:18:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:Suppressions used: 2024-08-16T02:18:55.942 INFO:teuthology.orchestra.run.smithi067.stderr: count bytes template 2024-08-16T02:18:55.942 INFO:teuthology.orchestra.run.smithi067.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:55.942 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-08-16T02:18:55.998 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:56.002 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:56,002 seastar - Reactor backend: linux-aio 2024-08-16T02:18:56.010 INFO:teuthology.orchestra.run.smithi067.stderr:==29636==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113ca1 0x4a0bd 0x35b7c 0x3689e 0x413dea1 0x5f4f424 0x5f4f97a 0x5d929a1 0x5daabe3 0x70ee632 0x70ffe05 0x71012a2 0x7101327 0x7101429 0x7101521 0x710172e 0x7101914 0x7101c63 0x7102007 0x7102237 0x71024a1 0x710276e 0x710298c 0x7102a44 0x7102b0a 0x7102beb 0x5bfe1b8 0x5bfe2c2 0x5bfe37e 0x5bfe43a 0x5bfe4f6 0x5bfe643 0x5bfe7ce 0x5bfeb8f 0x5c1aad0 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:56.290 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x3a35014 0x3afc9af 0xbac630f 0xbac75e2 0xbac785c 0xbac7a04 0xbac97bf 0xbac9c0a 0xbab5fba 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:56.290 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:56.492 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113c62 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:56.550 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x9e4b376 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:56.550 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bcd 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x4a0c5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:56.658 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d82e5 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:56.658 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113c19 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:57.016 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x2a590 0xddd2d 0x36e7625 0xbab6e51 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:57.016 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8841 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:Suppressions used: 2024-08-16T02:18:58.149 INFO:teuthology.orchestra.run.smithi067.stderr: count bytes template 2024-08-16T02:18:58.149 INFO:teuthology.orchestra.run.smithi067.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:58.149 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-08-16T02:18:58.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-16T02:18:58.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:18:58.525 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:58.526 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:58,527 seastar - Reactor backend: linux-aio 2024-08-16T02:18:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:==29683==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:58.637 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x5f807ae 0x5f809f5 0x5fcf0f3 0x5fcf2f9 0x5f4f446 0x5f4f620 0x5f4fbac 0x5d929a1 0x5daabe3 0x70ee632 0x70ffe05 0x71012a2 0x7101327 0x7101429 0x7101521 0x710172e 0x7101914 0x7101c63 0x7102007 0x7102237 0x71024a1 0x710276e 0x710298c 0x7102a44 0x7102b0a 0x7102beb 0x5bfe1b8 0x5bfe2c2 0x5bfe37e 0x5bfe43a 0x5bfe4f6 0x5bfe643 0x5bfe7ce 0x5bfeb8f 0x5c1aad0 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x38e8d78 0x38e98d1 0x3788ab2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:58.637 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:58,650 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:18:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:58,650 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-08-16T02:18:58.651 INFO:teuthology.orchestra.run.smithi067.stderr:DEBUG 2024-08-16 02:18:58,651 [shard 0:main] seastar - reactor::drain 2024-08-16T02:18:59.208 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:59.208 INFO:teuthology.orchestra.run.smithi067.stderr:Suppressions used: 2024-08-16T02:18:59.208 INFO:teuthology.orchestra.run.smithi067.stderr: count bytes template 2024-08-16T02:18:59.208 INFO:teuthology.orchestra.run.smithi067.stderr: 1 8 MallocExtension::Register 2024-08-16T02:18:59.209 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:18:59.209 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-08-16T02:18:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:18:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:INFO 2024-08-16 02:18:59,271 seastar - Reactor backend: linux-aio 2024-08-16T02:18:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:==29681==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:18:59.343 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x413dcc3 0x5f4f424 0x5f4f97a 0x5d929a1 0x5daabe3 0x70ee632 0x70ffe05 0x71012a2 0x7101327 0x7101429 0x7101521 0x710172e 0x7101914 0x7101c63 0x7102007 0x7102237 0x71024a1 0x710276e 0x710298c 0x7102a44 0x7102b0a 0x7102beb 0x5bfe1b8 0x5bfe2c2 0x5bfe37e 0x5bfe43a 0x5bfe4f6 0x5bfe643 0x5bfe7ce 0x5bfeb8f 0x5c1aad0 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:59.343 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x2d6c8 0x29edb 0xdfdb3 0x95a3a51 0x95a46a8 0x95a4b71 0x9ae59ab 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:59.556 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:59.757 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113cb0 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:59.757 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:59.814 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x8b51c 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:59.815 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:59.857 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d889b 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:59.858 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:59.915 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d88f3 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:18:59.915 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:18:59.922 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113c98 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:18:59.922 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:19:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7a04 0x54d89ce 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:19:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:19:00.295 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xc9b73 0xca7a6 0x35920f0 0x3afc7c2 0x3afc9ba 0xbadb0c0 0xbb1555f 0xbb158ea 0xbab6dd6 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:19:00.295 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:19:00.403 INFO:teuthology.orchestra.run.smithi067.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8a1a 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:19:00.403 INFO:teuthology.orchestra.run.smithi067.stderr:kernel callstack: 2024-08-16T02:19:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:19:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:Suppressions used: 2024-08-16T02:19:01.415 INFO:teuthology.orchestra.run.smithi067.stderr: count bytes template 2024-08-16T02:19:01.415 INFO:teuthology.orchestra.run.smithi067.stderr: 1 8 MallocExtension::Register 2024-08-16T02:19:01.415 INFO:teuthology.orchestra.run.smithi067.stderr:----------------------------------------------------- 2024-08-16T02:19:01.415 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-08-16T02:19:01.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-16T02:19:01.471 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-16T02:19:01.472 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:19:01.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-16T02:19:01.537 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:19:01.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-16T02:19:01.603 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:19:01.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-16T02:19:01.668 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:19:01.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-16T02:19:01.734 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:19:01.734 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-16T02:19:01.789 INFO:tasks.ceph:Adding keys to all mons... 2024-08-16T02:19:01.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAJt75mevEBCRAAO7P8g5jedDdAVsNMhFaJKw== 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAMt75mYHQCNBAAVaBohJRx5xFQfKZKcEEJnw== 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAQt75mVQcSCRAAbhFqxAoKRhc0nxDfrjw4Zw== 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQATt75mcY8DGRAAGlDDWOKwwbXyQeHor/Dt8Q== 2024-08-16T02:19:01.853 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-08-16T02:19:01.854 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAJt75meKotDBAAhii9WDQh2HZoK8P1Ypim1w== 2024-08-16T02:19:01.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-16T02:19:01.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:19:02.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:19:02.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:19:02.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-16T02:19:02.206 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-16T02:19:02.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-16T02:19:02.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-16T02:19:02.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:02.367+0000 7f56338fcc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:02.367+0000 7f56338fcc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-16T02:19:02.407 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:19:02.463 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-16T02:19:02.464 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-16T02:19:02.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-16T02:19:02.506 INFO:tasks.ceph.mon.a:Started 2024-08-16T02:19:02.506 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-16T02:19:02.507 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-16T02:19:02.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-16T02:19:02.509 INFO:tasks.ceph.mgr.x:Started 2024-08-16T02:19:02.509 DEBUG:tasks.ceph:set 0 configs 2024-08-16T02:19:02.509 INFO:tasks.ceph:Setting crush tunables to default 2024-08-16T02:19:02.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-08-16T02:19:02.593 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-08-16T02:19:02.592+0000 7f0ef2e31c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.593 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-08-16T02:19:02.592+0000 7f0ef2e31c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.593 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-08-16T02:19:02.592+0000 7f0ef2e31c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.594 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-08-16T02:19:02.594+0000 7f05eb039000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:02.617+0000 7fb070912640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:02.629+0000 7fb070912640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.706 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-08-16T02:19:02.718 INFO:tasks.ceph:check_enable_crimson: True 2024-08-16T02:19:02.718 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-08-16T02:19:02.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-08-16T02:19:02.801 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-08-16T02:19:02.800+0000 7f05eb039000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.801 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-08-16T02:19:02.800+0000 7f05eb039000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:02.823+0000 7ff4098ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:02.833+0000 7ff4098ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:02.921 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-08-16T02:19:02.920+0000 7f0eeadf4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-08-16T02:19:02.921 INFO:tasks.ceph.mon.a.smithi067.stderr:Please be aware that this feature is experimental, untested, 2024-08-16T02:19:02.921 INFO:tasks.ceph.mon.a.smithi067.stderr:unsupported, and may result in data corruption, data loss, 2024-08-16T02:19:02.921 INFO:tasks.ceph.mon.a.smithi067.stderr:and/or irreparable damage to your cluster. Do not use 2024-08-16T02:19:02.921 INFO:tasks.ceph.mon.a.smithi067.stderr:feature with important data. 2024-08-16T02:19:02.922 INFO:tasks.ceph.mon.a.smithi067.stderr: 2024-08-16T02:19:03.719 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-16T02:19:03.719 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:19:03.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-16T02:19:03.744 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:19:03.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-16T02:19:03.811 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:19:03.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-16T02:19:03.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new a550ea25-2edb-4628-ab47-f1133850bc01 0 2024-08-16T02:19:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:04.024+0000 7f7984152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:04.034+0000 7f7984152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:04.121 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-08-16T02:19:04.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 8b75d572-7db0-40cf-9f39-9063c0eec5de 1 2024-08-16T02:19:04.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:04.238+0000 7f9c79217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:04.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:04.248+0000 7f9c79217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:04.335 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-08-16T02:19:04.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new e3556829-8f2e-4544-82f9-0ab185f41290 2 2024-08-16T02:19:04.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:04.452+0000 7fa1c2e57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:04.462+0000 7fa1c2e57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:04.547 INFO:teuthology.orchestra.run.smithi067.stdout:2 2024-08-16T02:19:04.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-08-16T02:19:04.589 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-16T02:19:04.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-08-16T02:19:04.632 INFO:tasks.ceph.osd.0:Started 2024-08-16T02:19:04.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-08-16T02:19:04.662 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-16T02:19:04.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-08-16T02:19:04.704 INFO:tasks.ceph.osd.1:Started 2024-08-16T02:19:04.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-08-16T02:19:04.736 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-16T02:19:04.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-08-16T02:19:04.779 INFO:tasks.ceph.osd.2:Started 2024-08-16T02:19:04.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:19:04.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:04.904+0000 7fbce2192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:04.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:04.914+0000 7fbce2192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:04.993 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:19:04.993 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":6,"fsid":"734e1993-fc74-4287-90f2-01825483bae0","created":"2024-08-16T02:19:02.617605+0000","modified":"2024-08-16T02:19:04.546626+0000","last_up_change":"0.000000","last_in_change":"2024-08-16T02:19:04.546626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a550ea25-2edb-4628-ab47-f1133850bc01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8b75d572-7db0-40cf-9f39-9063c0eec5de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3556829-8f2e-4544-82f9-0ab185f41290","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:19:05.007 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-16T02:19:05.007 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-16T02:19:05.309 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-16T02:19:05.420 INFO:teuthology.misc.health.smithi067.stderr:2024-08-16T02:19:05.415+0000 7f5024952640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:05.425 INFO:teuthology.misc.health.smithi067.stderr:2024-08-16T02:19:05.424+0000 7f5024952640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:05.499 INFO:teuthology.misc.health.smithi067.stdout: 2024-08-16T02:19:05.499 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":6,"fsid":"734e1993-fc74-4287-90f2-01825483bae0","created":"2024-08-16T02:19:02.617605+0000","modified":"2024-08-16T02:19:04.546626+0000","last_up_change":"0.000000","last_in_change":"2024-08-16T02:19:04.546626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a550ea25-2edb-4628-ab47-f1133850bc01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8b75d572-7db0-40cf-9f39-9063c0eec5de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3556829-8f2e-4544-82f9-0ab185f41290","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:19:05.509 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-16T02:19:08.527 INFO:tasks.ceph.mgr.x.smithi067.stderr:2024-08-16T02:19:08.526+0000 7f0588a1a640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-16T02:19:11.812 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-16T02:19:11.929 INFO:teuthology.misc.health.smithi067.stderr:2024-08-16T02:19:11.923+0000 7ff163392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:11.932 INFO:teuthology.misc.health.smithi067.stderr:2024-08-16T02:19:11.932+0000 7ff163392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:12.140 INFO:teuthology.misc.health.smithi067.stdout: 2024-08-16T02:19:12.141 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"734e1993-fc74-4287-90f2-01825483bae0","created":"2024-08-16T02:19:02.617605+0000","modified":"2024-08-16T02:19:11.588524+0000","last_up_change":"2024-08-16T02:19:10.579561+0000","last_in_change":"2024-08-16T02:19:04.546626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:19:11.541917+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a550ea25-2edb-4628-ab47-f1133850bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":4185550721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":4185550721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4185550721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":4185550721}]},"public_addr":"172.21.15.67:6808/4185550721","cluster_addr":"172.21.15.67:6809/4185550721","heartbeat_back_addr":"172.21.15.67:6806/4185550721","heartbeat_front_addr":"172.21.15.67:6812/4185550721","state":["exists","up"]},{"osd":1,"uuid":"8b75d572-7db0-40cf-9f39-9063c0eec5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":467805611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":467805611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":467805611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":467805611}]},"public_addr":"172.21.15.67:6803/467805611","cluster_addr":"172.21.15.67:6802/467805611","heartbeat_back_addr":"172.21.15.67:6805/467805611","heartbeat_front_addr":"172.21.15.67:6804/467805611","state":["exists","up"]},{"osd":2,"uuid":"e3556829-8f2e-4544-82f9-0ab185f41290","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3162681119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3162681119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3162681119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":3162681119}]},"public_addr":"172.21.15.67:6810/3162681119","cluster_addr":"172.21.15.67:6811/3162681119","heartbeat_back_addr":"172.21.15.67:6816/3162681119","heartbeat_front_addr":"172.21.15.67:6815/3162681119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:19:12.152 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-08-16T02:19:12.152 INFO:tasks.ceph:Creating RBD pool 2024-08-16T02:19:12.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-16T02:19:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:12.257+0000 7f46bd352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:12.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:12.268+0000 7f46bd352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:12.596 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'rbd' created 2024-08-16T02:19:12.614 DEBUG:teuthology.orchestra.run.smithi067:> rbd --cluster ceph pool init rbd 2024-08-16T02:19:12.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:12.714+0000 7fda3db0c8c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:12.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:12.714+0000 7fda3db0c8c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:12.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:12.722+0000 7fda3db0c8c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:15.678 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-16T02:19:15.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-08-16T02:19:15.679 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-16T02:19:15.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:15.790+0000 7f9ce8a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:15.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:15.799+0000 7f9ce8a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:19:16.017 INFO:teuthology.orchestra.run.smithi067.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-08-16T02:19:16.018 INFO:tasks.ceph_manager:config epoch is 1 2024-08-16T02:19:16.018 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-16T02:19:16.018 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-16T02:19:16.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-16T02:19:16.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:16.127+0000 7f3b09192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:16.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:16.135+0000 7f3b09192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:16.396 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:19:16.407 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":4010660755},{"type":"v1","addr":"172.21.15.67:6801","nonce":4010660755}]},"active_addr":"172.21.15.67:6801/4010660755","active_change":"2024-08-16T02:19:07.513470+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3820075480}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":395836810}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2722673942}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1957615281}]}]} 2024-08-16T02:19:16.411 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-16T02:19:16.411 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-16T02:19:16.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:19:16.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:16.519+0000 7f5bb8e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:16.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:16.527+0000 7f5bb8e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:16.737 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:19:16.737 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":14,"fsid":"734e1993-fc74-4287-90f2-01825483bae0","created":"2024-08-16T02:19:02.617605+0000","modified":"2024-08-16T02:19:15.606103+0000","last_up_change":"2024-08-16T02:19:10.579561+0000","last_in_change":"2024-08-16T02:19:04.546626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:19:11.541917+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:19:12.476276+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a550ea25-2edb-4628-ab47-f1133850bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":4185550721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":4185550721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4185550721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":4185550721}]},"public_addr":"172.21.15.67:6808/4185550721","cluster_addr":"172.21.15.67:6809/4185550721","heartbeat_back_addr":"172.21.15.67:6806/4185550721","heartbeat_front_addr":"172.21.15.67:6812/4185550721","state":["exists","up"]},{"osd":1,"uuid":"8b75d572-7db0-40cf-9f39-9063c0eec5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":467805611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":467805611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":467805611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":467805611}]},"public_addr":"172.21.15.67:6803/467805611","cluster_addr":"172.21.15.67:6802/467805611","heartbeat_back_addr":"172.21.15.67:6805/467805611","heartbeat_front_addr":"172.21.15.67:6804/467805611","state":["exists","up"]},{"osd":2,"uuid":"e3556829-8f2e-4544-82f9-0ab185f41290","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3162681119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3162681119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3162681119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":3162681119}]},"public_addr":"172.21.15.67:6810/3162681119","cluster_addr":"172.21.15.67:6811/3162681119","heartbeat_back_addr":"172.21.15.67:6816/3162681119","heartbeat_front_addr":"172.21.15.67:6815/3162681119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:19:16.750 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-16T02:19:16.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:19:16.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:16.859+0000 7fb66e42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:16.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:16.867+0000 7fb66e42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:19:17.075 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":14,"fsid":"734e1993-fc74-4287-90f2-01825483bae0","created":"2024-08-16T02:19:02.617605+0000","modified":"2024-08-16T02:19:15.606103+0000","last_up_change":"2024-08-16T02:19:10.579561+0000","last_in_change":"2024-08-16T02:19:04.546626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:19:11.541917+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:19:12.476276+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a550ea25-2edb-4628-ab47-f1133850bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":4185550721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":4185550721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4185550721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":4185550721}]},"public_addr":"172.21.15.67:6808/4185550721","cluster_addr":"172.21.15.67:6809/4185550721","heartbeat_back_addr":"172.21.15.67:6806/4185550721","heartbeat_front_addr":"172.21.15.67:6812/4185550721","state":["exists","up"]},{"osd":1,"uuid":"8b75d572-7db0-40cf-9f39-9063c0eec5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":467805611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":467805611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":467805611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":467805611}]},"public_addr":"172.21.15.67:6803/467805611","cluster_addr":"172.21.15.67:6802/467805611","heartbeat_back_addr":"172.21.15.67:6805/467805611","heartbeat_front_addr":"172.21.15.67:6804/467805611","state":["exists","up"]},{"osd":2,"uuid":"e3556829-8f2e-4544-82f9-0ab185f41290","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3162681119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3162681119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3162681119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":3162681119}]},"public_addr":"172.21.15.67:6810/3162681119","cluster_addr":"172.21.15.67:6811/3162681119","heartbeat_back_addr":"172.21.15.67:6816/3162681119","heartbeat_front_addr":"172.21.15.67:6815/3162681119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:19:17.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-16T02:19:17.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-16T02:19:17.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-16T02:19:17.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.202+0000 7f80f6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.202+0000 7fd957592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.203+0000 7fb0d50ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.213+0000 7fd957592640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.213+0000 7f80f6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.213+0000 7fb0d50ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.247 INFO:teuthology.orchestra.run.smithi067.stdout:38654705670 2024-08-16T02:19:17.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-16T02:19:17.248 INFO:teuthology.orchestra.run.smithi067.stdout:34359738375 2024-08-16T02:19:17.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-16T02:19:17.249 INFO:teuthology.orchestra.run.smithi067.stdout:38654705670 2024-08-16T02:19:17.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-16T02:19:17.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.402+0000 7f57f0752640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.402+0000 7f2705e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.403+0000 7fc79c152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.412+0000 7f57f0752640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.413+0000 7f2705e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:17.413+0000 7fc79c152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:17.626 INFO:teuthology.orchestra.run.smithi067.stdout:38654705669 2024-08-16T02:19:17.637 INFO:teuthology.orchestra.run.smithi067.stdout:34359738375 2024-08-16T02:19:17.638 INFO:teuthology.orchestra.run.smithi067.stdout:38654705670 2024-08-16T02:19:17.640 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-08-16T02:19:17.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2024-08-16T02:19:17.652 DEBUG:teuthology.parallel:result is None 2024-08-16T02:19:17.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2024-08-16T02:19:17.653 DEBUG:teuthology.parallel:result is None 2024-08-16T02:19:18.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-16T02:19:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:18.764+0000 7f8c6046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:18.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:18.773+0000 7f8c6046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:18.982 INFO:teuthology.orchestra.run.smithi067.stdout:38654705670 2024-08-16T02:19:18.995 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.0 2024-08-16T02:19:18.996 DEBUG:teuthology.parallel:result is None 2024-08-16T02:19:18.996 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-16T02:19:18.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:19:19.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:19.105+0000 7f0bd7b92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:19.114+0000 7f0bd7b92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:19.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:19:19.318 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-08-16T02:19:19.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-08-16T02:19:17.519508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3348,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889836,"statfs":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.285110"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.635489+0000","last_change":"2024-08-16T02:19:15.641320+0000","last_active":"2024-08-16T02:19:15.635489+0000","last_peered":"2024-08-16T02:19:15.635489+0000","last_clean":"2024-08-16T02:19:15.635489+0000","last_became_active":"2024-08-16T02:19:13.709659+0000","last_became_peered":"2024-08-16T02:19:13.709659+0000","last_unstale":"2024-08-16T02:19:15.635489+0000","last_undegraded":"2024-08-16T02:19:15.635489+0000","last_fullsized":"2024-08-16T02:19:15.635489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.638603+0000","last_change":"2024-08-16T02:19:15.647014+0000","last_active":"2024-08-16T02:19:15.638603+0000","last_peered":"2024-08-16T02:19:15.638603+0000","last_clean":"2024-08-16T02:19:15.638603+0000","last_became_active":"2024-08-16T02:19:13.799083+0000","last_became_peered":"2024-08-16T02:19:13.799083+0000","last_unstale":"2024-08-16T02:19:15.638603+0000","last_undegraded":"2024-08-16T02:19:15.638603+0000","last_fullsized":"2024-08-16T02:19:15.638603+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.629953+0000","last_change":"2024-08-16T02:19:15.640509+0000","last_active":"2024-08-16T02:19:15.629953+0000","last_peered":"2024-08-16T02:19:15.629953+0000","last_clean":"2024-08-16T02:19:15.629953+0000","last_became_active":"2024-08-16T02:19:13.722578+0000","last_became_peered":"2024-08-16T02:19:13.722578+0000","last_unstale":"2024-08-16T02:19:15.629953+0000","last_undegraded":"2024-08-16T02:19:15.629953+0000","last_fullsized":"2024-08-16T02:19:15.629953+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.637347+0000","last_change":"2024-08-16T02:19:15.648434+0000","last_active":"2024-08-16T02:19:15.637347+0000","last_peered":"2024-08-16T02:19:15.637347+0000","last_clean":"2024-08-16T02:19:15.637347+0000","last_became_active":"2024-08-16T02:19:13.794874+0000","last_became_peered":"2024-08-16T02:19:13.794874+0000","last_unstale":"2024-08-16T02:19:15.637347+0000","last_undegraded":"2024-08-16T02:19:15.637347+0000","last_fullsized":"2024-08-16T02:19:15.637347+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.656601+0000","last_change":"2024-08-16T02:19:15.656601+0000","last_active":"2024-08-16T02:19:15.656601+0000","last_peered":"2024-08-16T02:19:15.656601+0000","last_clean":"2024-08-16T02:19:15.656601+0000","last_became_active":"2024-08-16T02:19:13.807895+0000","last_became_peered":"2024-08-16T02:19:13.807895+0000","last_unstale":"2024-08-16T02:19:15.656601+0000","last_undegraded":"2024-08-16T02:19:15.656601+0000","last_fullsized":"2024-08-16T02:19:15.656601+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.648742+0000","last_change":"2024-08-16T02:19:15.653108+0000","last_active":"2024-08-16T02:19:15.648742+0000","last_peered":"2024-08-16T02:19:15.648742+0000","last_clean":"2024-08-16T02:19:15.648742+0000","last_became_active":"2024-08-16T02:19:13.747889+0000","last_became_peered":"2024-08-16T02:19:13.747889+0000","last_unstale":"2024-08-16T02:19:15.648742+0000","last_undegraded":"2024-08-16T02:19:15.648742+0000","last_fullsized":"2024-08-16T02:19:15.648742+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.642055+0000","last_change":"2024-08-16T02:19:15.647826+0000","last_active":"2024-08-16T02:19:15.642055+0000","last_peered":"2024-08-16T02:19:15.642055+0000","last_clean":"2024-08-16T02:19:15.642055+0000","last_became_active":"2024-08-16T02:19:13.820022+0000","last_became_peered":"2024-08-16T02:19:13.820022+0000","last_unstale":"2024-08-16T02:19:15.642055+0000","last_undegraded":"2024-08-16T02:19:15.642055+0000","last_fullsized":"2024-08-16T02:19:15.642055+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.640089+0000","last_change":"2024-08-16T02:19:15.648266+0000","last_active":"2024-08-16T02:19:15.640089+0000","last_peered":"2024-08-16T02:19:15.640089+0000","last_clean":"2024-08-16T02:19:15.640089+0000","last_became_active":"2024-08-16T02:19:13.719057+0000","last_became_peered":"2024-08-16T02:19:13.719057+0000","last_unstale":"2024-08-16T02:19:15.640089+0000","last_undegraded":"2024-08-16T02:19:15.640089+0000","last_fullsized":"2024-08-16T02:19:15.640089+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.633126+0000","last_change":"2024-08-16T02:19:12.699715+0000","last_active":"2024-08-16T02:19:15.633126+0000","last_peered":"2024-08-16T02:19:15.633126+0000","last_clean":"2024-08-16T02:19:15.633126+0000","last_became_active":"2024-08-16T02:19:12.660251+0000","last_became_peered":"2024-08-16T02:19:12.660251+0000","last_unstale":"2024-08-16T02:19:15.633126+0000","last_undegraded":"2024-08-16T02:19:15.633126+0000","last_fullsized":"2024-08-16T02:19:15.633126+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_clean_scrub_stamp":"2024-08-16T02:19:11.588524+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":428,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297300,"statfs":{"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1588,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296140,"statfs":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296396,"statfs":{"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:19:19.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:19:19.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:19.440+0000 7f82b6152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:19.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:19.448+0000 7f82b6152640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:19.659 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:19:19.659 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-08-16T02:19:19.672 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-08-16T02:19:19.519923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3348,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889836,"statfs":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.285524"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.635489+0000","last_change":"2024-08-16T02:19:15.641320+0000","last_active":"2024-08-16T02:19:15.635489+0000","last_peered":"2024-08-16T02:19:15.635489+0000","last_clean":"2024-08-16T02:19:15.635489+0000","last_became_active":"2024-08-16T02:19:13.709659+0000","last_became_peered":"2024-08-16T02:19:13.709659+0000","last_unstale":"2024-08-16T02:19:15.635489+0000","last_undegraded":"2024-08-16T02:19:15.635489+0000","last_fullsized":"2024-08-16T02:19:15.635489+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.638603+0000","last_change":"2024-08-16T02:19:15.647014+0000","last_active":"2024-08-16T02:19:15.638603+0000","last_peered":"2024-08-16T02:19:15.638603+0000","last_clean":"2024-08-16T02:19:15.638603+0000","last_became_active":"2024-08-16T02:19:13.799083+0000","last_became_peered":"2024-08-16T02:19:13.799083+0000","last_unstale":"2024-08-16T02:19:15.638603+0000","last_undegraded":"2024-08-16T02:19:15.638603+0000","last_fullsized":"2024-08-16T02:19:15.638603+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.629953+0000","last_change":"2024-08-16T02:19:15.640509+0000","last_active":"2024-08-16T02:19:15.629953+0000","last_peered":"2024-08-16T02:19:15.629953+0000","last_clean":"2024-08-16T02:19:15.629953+0000","last_became_active":"2024-08-16T02:19:13.722578+0000","last_became_peered":"2024-08-16T02:19:13.722578+0000","last_unstale":"2024-08-16T02:19:15.629953+0000","last_undegraded":"2024-08-16T02:19:15.629953+0000","last_fullsized":"2024-08-16T02:19:15.629953+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.637347+0000","last_change":"2024-08-16T02:19:15.648434+0000","last_active":"2024-08-16T02:19:15.637347+0000","last_peered":"2024-08-16T02:19:15.637347+0000","last_clean":"2024-08-16T02:19:15.637347+0000","last_became_active":"2024-08-16T02:19:13.794874+0000","last_became_peered":"2024-08-16T02:19:13.794874+0000","last_unstale":"2024-08-16T02:19:15.637347+0000","last_undegraded":"2024-08-16T02:19:15.637347+0000","last_fullsized":"2024-08-16T02:19:15.637347+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'1","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.656601+0000","last_change":"2024-08-16T02:19:15.656601+0000","last_active":"2024-08-16T02:19:15.656601+0000","last_peered":"2024-08-16T02:19:15.656601+0000","last_clean":"2024-08-16T02:19:15.656601+0000","last_became_active":"2024-08-16T02:19:13.807895+0000","last_became_peered":"2024-08-16T02:19:13.807895+0000","last_unstale":"2024-08-16T02:19:15.656601+0000","last_undegraded":"2024-08-16T02:19:15.656601+0000","last_fullsized":"2024-08-16T02:19:15.656601+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.648742+0000","last_change":"2024-08-16T02:19:15.653108+0000","last_active":"2024-08-16T02:19:15.648742+0000","last_peered":"2024-08-16T02:19:15.648742+0000","last_clean":"2024-08-16T02:19:15.648742+0000","last_became_active":"2024-08-16T02:19:13.747889+0000","last_became_peered":"2024-08-16T02:19:13.747889+0000","last_unstale":"2024-08-16T02:19:15.648742+0000","last_undegraded":"2024-08-16T02:19:15.648742+0000","last_fullsized":"2024-08-16T02:19:15.648742+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.642055+0000","last_change":"2024-08-16T02:19:15.647826+0000","last_active":"2024-08-16T02:19:15.642055+0000","last_peered":"2024-08-16T02:19:15.642055+0000","last_clean":"2024-08-16T02:19:15.642055+0000","last_became_active":"2024-08-16T02:19:13.820022+0000","last_became_peered":"2024-08-16T02:19:13.820022+0000","last_unstale":"2024-08-16T02:19:15.642055+0000","last_undegraded":"2024-08-16T02:19:15.642055+0000","last_fullsized":"2024-08-16T02:19:15.642055+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.640089+0000","last_change":"2024-08-16T02:19:15.648266+0000","last_active":"2024-08-16T02:19:15.640089+0000","last_peered":"2024-08-16T02:19:15.640089+0000","last_clean":"2024-08-16T02:19:15.640089+0000","last_became_active":"2024-08-16T02:19:13.719057+0000","last_became_peered":"2024-08-16T02:19:13.719057+0000","last_unstale":"2024-08-16T02:19:15.640089+0000","last_undegraded":"2024-08-16T02:19:15.640089+0000","last_fullsized":"2024-08-16T02:19:15.640089+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":84,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-08-16T02:19:15.633126+0000","last_change":"2024-08-16T02:19:12.699715+0000","last_active":"2024-08-16T02:19:15.633126+0000","last_peered":"2024-08-16T02:19:15.633126+0000","last_clean":"2024-08-16T02:19:15.633126+0000","last_became_active":"2024-08-16T02:19:12.660251+0000","last_became_peered":"2024-08-16T02:19:12.660251+0000","last_unstale":"2024-08-16T02:19:15.633126+0000","last_undegraded":"2024-08-16T02:19:15.633126+0000","last_fullsized":"2024-08-16T02:19:15.633126+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_clean_scrub_stamp":"2024-08-16T02:19:11.588524+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655192064,"internally_reserved":0,"allocated":3428352,"data_stored":3428352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":428,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297300,"statfs":{"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1588,"kb_used_data":1588,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296140,"statfs":{"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296396,"statfs":{"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551509504,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551247360,"internally_reserved":0,"allocated":1626112,"data_stored":1626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552435200,"internally_reserved":0,"allocated":438272,"data_stored":438272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:19:19.672 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-16T02:19:19.673 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-16T02:19:19.673 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-16T02:19:19.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-16T02:19:19.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:19.781+0000 7ff7ab8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:19.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:19:19.788+0000 7ff7ab8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:19:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:19:20.035 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-16T02:19:20.048 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-16T02:19:20.048 INFO:teuthology.run_tasks:Running task workunit... 2024-08-16T02:19:20.063 INFO:tasks.workunit:Pulling workunits from ref 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:19:20.063 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-16T02:19:20.064 DEBUG:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-16T02:19:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:19:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-16T02:19:20.080 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-16T02:19:20.135 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-16T02:19:20.135 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-16T02:19:20.191 INFO:tasks.workunit:timeout=3h 2024-08-16T02:19:20.191 INFO:tasks.workunit:cleanup=True 2024-08-16T02:19:20.192 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:19:20.248 INFO:tasks.workunit.client.0.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-16T02:20:19.483 INFO:tasks.workunit.client.0.smithi067.stderr:Updating files: 91% (11601/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-08-16T02:20:20.148 INFO:tasks.workunit.client.0.smithi067.stderr:Note: switching to '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'. 2024-08-16T02:20:20.148 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-08-16T02:20:20.148 INFO:tasks.workunit.client.0.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-16T02:20:20.148 INFO:tasks.workunit.client.0.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-16T02:20:20.148 INFO:tasks.workunit.client.0.smithi067.stderr:state without impacting any branches by switching back to a branch. 2024-08-16T02:20:20.148 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-08-16T02:20:20.148 INFO:tasks.workunit.client.0.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr: git switch -c 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr:Or undo this operation with: 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr: git switch - 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-08-16T02:20:20.149 INFO:tasks.workunit.client.0.smithi067.stderr:HEAD is now at 6f386a3f0b1 Merge branch 'wip-seastore-more-reports' of https://github.com/cyx1231st/ceph into wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:20:20.152 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-16T02:20:20.215 INFO:tasks.workunit.client.0.smithi067.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-16T02:20:20.218 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-16T02:20:20.218 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-16T02:20:20.820 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-16T02:20:20.878 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-16T02:20:20.921 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-16T02:20:20.922 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-16T02:20:20.922 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-16T02:20:20.963 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-16T02:20:20.967 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-08-16T02:20:20.967 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-16T02:20:21.024 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2024-08-16T02:20:21.025 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2024-08-16T02:20:21.025 DEBUG:teuthology.orchestra.run.smithi067:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-08-16T02:20:21.132 INFO:tasks.workunit.client.0.smithi067.stdout:seed 31600 2024-08-16T02:20:21.132 INFO:tasks.workunit.client.0.smithi067.stdout:Note: Google Test filter = TestLibRBD.* 2024-08-16T02:20:21.132 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] Running 171 tests from 1 test suite. 2024-08-16T02:20:21.132 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment set-up. 2024-08-16T02:20:21.132 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 171 tests from TestLibRBD 2024-08-16T02:20:21.182 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-08-16T02:20:22.671 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:24.843 INFO:tasks.workunit.client.0.smithi067.stdout:image has size 2097152 and order 22 2024-08-16T02:20:24.843 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.CreateAndStat (3660 ms) 2024-08-16T02:20:24.843 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-08-16T02:20:24.843 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:780: Skipped 2024-08-16T02:20:24.843 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:24.844 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool (0 ms) 2024-08-16T02:20:24.844 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-08-16T02:20:24.844 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:24.944 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.CreateAndStatPP (101 ms) 2024-08-16T02:20:24.944 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.GetId 2024-08-16T02:20:24.944 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:25.046 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.GetId (102 ms) 2024-08-16T02:20:25.046 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-08-16T02:20:25.046 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:25.162 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.GetIdPP (116 ms) 2024-08-16T02:20:25.162 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-08-16T02:20:25.162 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:25.260 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (98 ms) 2024-08-16T02:20:25.260 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-08-16T02:20:25.260 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (100 ms) 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:939: Skipped 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp (0 ms) 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:962: Skipped 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:25.360 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP (0 ms) 2024-08-16T02:20:25.361 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.OpenAio 2024-08-16T02:20:25.361 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:25.460 INFO:tasks.workunit.client.0.smithi067.stdout:image has size 2097152 and order 22 2024-08-16T02:20:25.461 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.OpenAio (101 ms) 2024-08-16T02:20:25.461 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-08-16T02:20:25.473 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.OpenAioFail (12 ms) 2024-08-16T02:20:25.473 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-08-16T02:20:25.473 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:25.631 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.OpenAioPP (159 ms) 2024-08-16T02:20:25.632 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-08-16T02:20:25.644 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.OpenAioFailPP (12 ms) 2024-08-16T02:20:25.644 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-08-16T02:20:25.644 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:25.907 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.ResizeAndStat (262 ms) 2024-08-16T02:20:25.907 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-08-16T02:20:25.908 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:26.098 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (191 ms) 2024-08-16T02:20:26.099 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-08-16T02:20:26.099 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:26.327 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (229 ms) 2024-08-16T02:20:26.327 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-08-16T02:20:26.327 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:26.558 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (231 ms) 2024-08-16T02:20:26.558 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-08-16T02:20:27.989 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:29.854 INFO:tasks.workunit.client.0.smithi067.stdout:image: image15 2024-08-16T02:20:29.855 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image15 2024-08-16T02:20:29.855 INFO:tasks.workunit.client.0.smithi067.stdout:found image15 2024-08-16T02:20:29.855 INFO:tasks.workunit.client.0.smithi067.stdout:erased image15 2024-08-16T02:20:29.855 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:30.119 INFO:tasks.workunit.client.0.smithi067.stdout:image: image15 2024-08-16T02:20:30.119 INFO:tasks.workunit.client.0.smithi067.stdout:image: image16 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image15 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:found image15 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:erased image15 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image16 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:found image16 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:erased image16 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:image: image16 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image16 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:found image16 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:erased image16 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (3561 ms) 2024-08-16T02:20:30.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-08-16T02:20:31.728 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:33.900 INFO:tasks.workunit.client.0.smithi067.stdout:num images is: 1 2024-08-16T02:20:33.901 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:20:33.901 INFO:tasks.workunit.client.0.smithi067.stdout:image: image17 2024-08-16T02:20:33.901 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image17 2024-08-16T02:20:33.901 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:33.962 INFO:tasks.workunit.client.0.smithi067.stdout:num images is: 2 2024-08-16T02:20:33.962 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 2 2024-08-16T02:20:33.962 INFO:tasks.workunit.client.0.smithi067.stdout:image: image17 2024-08-16T02:20:33.962 INFO:tasks.workunit.client.0.smithi067.stdout:image: image18 2024-08-16T02:20:33.962 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image17 2024-08-16T02:20:33.962 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image18 2024-08-16T02:20:34.094 INFO:tasks.workunit.client.0.smithi067.stdout:num images is: 1 2024-08-16T02:20:34.094 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:20:34.094 INFO:tasks.workunit.client.0.smithi067.stdout:image: image18 2024-08-16T02:20:34.094 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image18 2024-08-16T02:20:34.094 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (3975 ms) 2024-08-16T02:20:34.094 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCopy 2024-08-16T02:20:35.750 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:44.783 INFO:tasks.workunit.client.0.smithi067.stdout:image: image19 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image19 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:found image19 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:erased image19 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:image: image19 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:image: image20 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image19 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:found image19 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:erased image19 2024-08-16T02:20:44.784 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image20 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:found image20 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:erased image20 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:0.00% done 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:100.00% done 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:image: image19 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:image: image20 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:image: image21 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image19 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:found image19 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:erased image19 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image20 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:found image20 2024-08-16T02:20:44.785 INFO:tasks.workunit.client.0.smithi067.stdout:erased image20 2024-08-16T02:20:44.786 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image21 2024-08-16T02:20:44.786 INFO:tasks.workunit.client.0.smithi067.stdout:found image21 2024-08-16T02:20:44.786 INFO:tasks.workunit.client.0.smithi067.stdout:erased image21 2024-08-16T02:20:44.786 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCopy (10690 ms) 2024-08-16T02:20:44.786 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-08-16T02:20:46.792 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:20:51.867 INFO:tasks.workunit.client.0.smithi067.stdout:num images is: 1 2024-08-16T02:20:51.867 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:20:51.867 INFO:tasks.workunit.client.0.smithi067.stdout:image: image22 2024-08-16T02:20:51.868 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image22 2024-08-16T02:20:54.258 INFO:tasks.workunit.client.0.smithi067.stdout:num images is: 2 2024-08-16T02:20:54.258 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 2 2024-08-16T02:20:54.258 INFO:tasks.workunit.client.0.smithi067.stdout:image: image22 2024-08-16T02:20:54.258 INFO:tasks.workunit.client.0.smithi067.stdout:image: image23 2024-08-16T02:20:54.258 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image22 2024-08-16T02:20:54.258 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image23 2024-08-16T02:20:55.111 INFO:tasks.workunit.client.0.smithi067.stdout:0.00% done 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:100.00% done 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:num images is: 3 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 3 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:image: image22 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:image: image23 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:image: image24 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image22 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image23 2024-08-16T02:20:55.112 INFO:tasks.workunit.client.0.smithi067.stdout:expected = image24 2024-08-16T02:20:55.735 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCopyPP (10951 ms) 2024-08-16T02:20:55.735 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:1578: Skipped 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy (0 ms) 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:1745: Skipped 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP (0 ms) 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-08-16T02:20:55.736 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:00.070 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 1 2024-08-16T02:21:00.070 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:21:00.070 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap1 2024-08-16T02:21:00.070 INFO:tasks.workunit.client.0.smithi067.stdout:found snap1 with size 2097152 2024-08-16T02:21:00.070 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 2 2024-08-16T02:21:00.070 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 2 2024-08-16T02:21:00.070 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap1 2024-08-16T02:21:00.070 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap2 2024-08-16T02:21:00.071 INFO:tasks.workunit.client.0.smithi067.stdout:found snap1 with size 2097152 2024-08-16T02:21:00.071 INFO:tasks.workunit.client.0.smithi067.stdout:found snap2 with size 4194304 2024-08-16T02:21:00.071 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 1 2024-08-16T02:21:00.071 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:21:00.071 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap2 2024-08-16T02:21:00.071 INFO:tasks.workunit.client.0.smithi067.stdout:found snap2 with size 4194304 2024-08-16T02:21:00.071 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 0 2024-08-16T02:21:00.071 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 0 2024-08-16T02:21:00.072 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (4334 ms) 2024-08-16T02:21:00.072 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-08-16T02:21:00.072 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:1890: Skipped 2024-08-16T02:21:00.072 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:21:00.072 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp (0 ms) 2024-08-16T02:21:00.072 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-08-16T02:21:00.072 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:01.006 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 1 2024-08-16T02:21:01.006 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:21:01.006 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap1 2024-08-16T02:21:01.006 INFO:tasks.workunit.client.0.smithi067.stdout:found snap1 with size 2097152 2024-08-16T02:21:02.023 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 2 2024-08-16T02:21:02.023 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 2 2024-08-16T02:21:02.023 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap1 2024-08-16T02:21:02.023 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap2 2024-08-16T02:21:02.023 INFO:tasks.workunit.client.0.smithi067.stdout:found snap1 with size 2097152 2024-08-16T02:21:02.023 INFO:tasks.workunit.client.0.smithi067.stdout:found snap2 with size 4194304 2024-08-16T02:21:03.076 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 1 2024-08-16T02:21:03.076 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:21:03.076 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap2 2024-08-16T02:21:03.076 INFO:tasks.workunit.client.0.smithi067.stdout:found snap2 with size 4194304 2024-08-16T02:21:04.054 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 0 2024-08-16T02:21:04.054 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 0 2024-08-16T02:21:04.070 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4001 ms) 2024-08-16T02:21:04.070 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-08-16T02:21:04.070 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:10.065 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 0 2024-08-16T02:21:10.065 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 0 2024-08-16T02:21:10.079 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6009 ms) 2024-08-16T02:21:10.079 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-08-16T02:21:10.079 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:11.110 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 1 2024-08-16T02:21:11.111 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:21:11.111 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap1 2024-08-16T02:21:11.111 INFO:tasks.workunit.client.0.smithi067.stdout:found snap1 with size 2097152 2024-08-16T02:21:12.089 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 2 2024-08-16T02:21:12.089 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 2 2024-08-16T02:21:12.089 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap1 2024-08-16T02:21:12.089 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap2 2024-08-16T02:21:12.090 INFO:tasks.workunit.client.0.smithi067.stdout:found snap1 with size 2097152 2024-08-16T02:21:12.090 INFO:tasks.workunit.client.0.smithi067.stdout:found snap2 with size 4194304 2024-08-16T02:21:12.144 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 2 2024-08-16T02:21:12.144 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 2 2024-08-16T02:21:12.144 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap1-rename 2024-08-16T02:21:12.144 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap2 2024-08-16T02:21:12.144 INFO:tasks.workunit.client.0.smithi067.stdout:found snap1-rename with size 2097152 2024-08-16T02:21:12.144 INFO:tasks.workunit.client.0.smithi067.stdout:found snap2 with size 4194304 2024-08-16T02:21:13.153 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 1 2024-08-16T02:21:13.153 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:21:13.153 INFO:tasks.workunit.client.0.smithi067.stdout:snap: snap2-rename 2024-08-16T02:21:13.153 INFO:tasks.workunit.client.0.smithi067.stdout:found snap2-rename with size 4194304 2024-08-16T02:21:14.125 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 0 2024-08-16T02:21:14.125 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 0 2024-08-16T02:21:14.142 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (4063 ms) 2024-08-16T02:21:14.142 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-08-16T02:21:15.921 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:15.921 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:15.921 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:15.921 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:18.732 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (4590 ms) 2024-08-16T02:21:18.733 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-08-16T02:21:23.210 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:21:43.698 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.RemoveFullTry (24964 ms) 2024-08-16T02:21:43.698 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-08-16T02:21:43.698 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2227: Skipped 2024-08-16T02:21:43.698 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:21:43.698 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool (0 ms) 2024-08-16T02:21:43.698 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-08-16T02:21:43.698 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2247: Skipped 2024-08-16T02:21:43.699 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:21:43.699 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespace (0 ms) 2024-08-16T02:21:43.699 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-08-16T02:21:43.699 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2269: Skipped 2024-08-16T02:21:43.699 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:21:43.699 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespaceDataPool (0 ms) 2024-08-16T02:21:43.699 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestIO 2024-08-16T02:21:43.699 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:22:14.564 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.565 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.566 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:22:14.567 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.568 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:22:14.569 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.570 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:22:14.571 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:discard: 512~512 = 512 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:aio discard: 1536~512 = 0 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:22:14.572 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:22:14.573 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.574 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.575 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:22:14.576 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.577 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:14.578 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.579 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.580 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.901 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.901 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.901 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.901 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.901 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.901 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.901 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.902 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.903 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.905 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.906 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.907 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.908 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:22:40.910 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:22:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestIO (57204 ms) 2024-08-16T02:22:40.912 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-08-16T02:22:40.912 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:23:16.602 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:16.603 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.604 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:23:16.605 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.606 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.607 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.608 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.609 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.609 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.609 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.609 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.609 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:23:16.609 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:23:16.609 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:23:16.610 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:discard: 512~512 = 512 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:aio discard: 1536~512 = 0 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:23:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.614 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.615 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.616 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.617 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.618 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.619 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:16.623 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.623 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.623 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:16.623 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.623 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.623 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.623 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.135 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.135 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.135 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.135 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.135 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.136 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.137 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.138 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.139 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.140 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.141 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.142 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.143 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:49.144 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 4 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:read: 4 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:read: 4 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (68234 ms) 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-08-16T02:23:49.145 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:23:55.118 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:55.118 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:55.118 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:55.118 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:55.118 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:55.119 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:23:55.120 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: -84 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:495: Failure 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout:Expected equality of these values: 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout: len 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout: Which is: 512 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout: static_cast(written) 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout: Which is: 18446744073709551532 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:630: Failure 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout:Value of: passed 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout: Actual: false 2024-08-16T02:23:55.121 INFO:tasks.workunit.client.0.smithi067.stdout:Expected: true 2024-08-16T02:24:00.170 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 4 2024-08-16T02:24:00.170 INFO:tasks.workunit.client.0.smithi067.stdout:read: 4 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:read: 4 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 (11036 ms) 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2471: Skipped 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption (0 ms) 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2659: Skipped 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (0 ms) 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2735: Skipped 2024-08-16T02:24:00.171 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:24:00.172 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (0 ms) 2024-08-16T02:24:00.172 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-08-16T02:24:00.172 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:24:04.782 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4612 ms) 2024-08-16T02:24:04.782 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-08-16T02:24:04.782 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (6055 ms) 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2905: Skipped 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:2985: Skipped 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch (0 ms) 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-08-16T02:24:10.837 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:24:25.885 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.EncryptedResize (15047 ms) 2024-08-16T02:24:25.885 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-08-16T02:24:25.885 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:3202: Skipped 2024-08-16T02:24:25.885 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:24:25.885 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData (0 ms) 2024-08-16T02:24:25.885 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-08-16T02:24:25.885 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:24:52.955 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:24:52.955 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:24:52.955 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:24:52.955 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:24:52.955 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:24:52.955 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.956 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:24:52.957 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:24:52.958 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.959 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.960 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:24:52.961 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:discard: 512~512 = 512 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:aio discard: 1536~512 = 0 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.962 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:24:52.963 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.964 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.965 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.966 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.967 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.968 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:24:52.969 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.970 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:24:52.971 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.003 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.003 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.003 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.003 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.003 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.003 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.003 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.004 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.005 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.006 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.007 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.008 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.010 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:16.011 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (50118 ms) 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:3947: Skipped 2024-08-16T02:25:16.012 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:25:16.013 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO (0 ms) 2024-08-16T02:25:16.013 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-08-16T02:25:16.013 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:16.244 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (242 ms) 2024-08-16T02:25:16.245 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-08-16T02:25:16.245 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:16.480 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (236 ms) 2024-08-16T02:25:16.480 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-08-16T02:25:16.480 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:16.656 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (176 ms) 2024-08-16T02:25:16.656 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-08-16T02:25:16.656 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:16.841 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (185 ms) 2024-08-16T02:25:16.841 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-08-16T02:25:16.841 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:17.277 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (436 ms) 2024-08-16T02:25:17.277 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-08-16T02:25:17.277 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:17.735 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (458 ms) 2024-08-16T02:25:17.736 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-08-16T02:25:17.736 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:18.245 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (510 ms) 2024-08-16T02:25:18.245 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-08-16T02:25:18.245 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:18.892 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (647 ms) 2024-08-16T02:25:18.893 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-08-16T02:25:18.893 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:23.447 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (4555 ms) 2024-08-16T02:25:23.447 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-08-16T02:25:23.447 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:28.304 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (4857 ms) 2024-08-16T02:25:28.305 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-08-16T02:25:28.305 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:28.511 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (207 ms) 2024-08-16T02:25:28.511 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-08-16T02:25:28.511 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:28.700 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (189 ms) 2024-08-16T02:25:28.700 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-08-16T02:25:28.700 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:28.880 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (181 ms) 2024-08-16T02:25:28.881 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-08-16T02:25:28.881 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:29.031 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (150 ms) 2024-08-16T02:25:29.031 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-08-16T02:25:29.031 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:aio discard: 0~1048576 = 0 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:aio discard: 0~4194304 = 0 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:aio discard: 3145728~1048576 = 0 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (126 ms) 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestFUA 2024-08-16T02:25:29.157 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:29.718 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.718 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.718 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.718 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.719 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.720 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:29.721 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:29.722 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:29.723 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestFUA (561 ms) 2024-08-16T02:25:29.723 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-08-16T02:25:29.723 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:25:29.813 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.813 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.814 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:25:29.815 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:30.047 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:start compare and write 2024-08-16T02:25:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:compare and wrote: 512 2024-08-16T02:25:30.049 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.049 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:25:30.049 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:30.065 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:30.066 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:25:30.066 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.066 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:25:30.066 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:25:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:25:30.084 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:30.102 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:30.102 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:25:30.102 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.102 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:25:30.102 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:30.122 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:30.122 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:25:30.122 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.122 INFO:tasks.workunit.client.0.smithi067.stdout:started aio compare and write 2024-08-16T02:25:30.122 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:30.172 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:30.172 INFO:tasks.workunit.client.0.smithi067.stdout:finished aio compare and write 2024-08-16T02:25:30.172 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:30.172 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:30.173 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:30.173 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:30.173 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:30.173 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.173 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:30.173 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:30.178 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:30.178 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:25:30.178 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.179 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:30.179 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:30.184 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:30.184 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:25:30.184 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.184 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:30.185 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:30.190 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:30.190 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:25:30.190 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.191 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:30.191 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:discard: 512~512 2024-08-16T02:25:30.197 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:53.171 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:53.171 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:53.171 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:53.171 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:53.171 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:25:53.171 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:25:53.171 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.172 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.173 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:25:53.174 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.175 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.176 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:25:53.176 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.176 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.176 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:25:53.176 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.176 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.176 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.177 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:53.178 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:53.179 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:53.381 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:53.381 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:53.381 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.382 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.382 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:53.382 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:53.382 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:53.529 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:53.529 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:53.529 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:53.529 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:53.529 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:53.530 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:53.530 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.530 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:53.530 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:53.530 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:53.530 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:53.530 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:53.530 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:54.144 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:54.144 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:54.144 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:54.145 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:54.145 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:54.145 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:54.145 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:54.557 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:54.557 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:54.557 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:54.557 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:54.557 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:54.557 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:54.557 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:55.385 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:55.385 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:55.385 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:55.385 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:55.385 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:55.385 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:55.385 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:55.937 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:55.937 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:55.937 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:55.937 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:55.937 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:55.937 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:55.937 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:55.938 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:55.938 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:55.938 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:25:55.938 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:55.938 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:55.938 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:57.261 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:57.261 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:57.261 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:57.262 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:57.262 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:57.262 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:57.262 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:58.086 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:58.086 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:58.086 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:58.086 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:58.086 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:58.086 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:58.086 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:25:59.613 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:25:59.613 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:25:59.613 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:25:59.613 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:25:59.613 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:25:59.613 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:25:59.613 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:00.705 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:00.706 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:00.706 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:02.512 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:02.512 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:02.512 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:02.512 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:02.512 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:02.512 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:02.512 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:03.753 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:03.753 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:03.753 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:03.754 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:03.754 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:03.754 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:03.754 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:05.861 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:05.862 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:05.862 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:05.862 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:05.862 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:05.862 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:05.862 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:07.271 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:07.271 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:07.271 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:07.271 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:07.271 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:07.271 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:07.271 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:07.271 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:07.272 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:07.272 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:07.272 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:07.272 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:07.272 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:09.813 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:09.814 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:09.814 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:09.814 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:09.814 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:09.814 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:09.814 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:11.488 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:11.488 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:11.488 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:11.489 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:11.489 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:11.489 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:11.489 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:14.190 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:14.190 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:14.190 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:14.190 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:14.190 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:14.191 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:14.191 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:16.108 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:16.108 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:16.108 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestIOPP (46390 ms) 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-08-16T02:26:16.109 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:16.266 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (158 ms) 2024-08-16T02:26:16.266 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-08-16T02:26:16.267 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:16.467 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (201 ms) 2024-08-16T02:26:16.468 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-08-16T02:26:16.468 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:16.612 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (145 ms) 2024-08-16T02:26:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-08-16T02:26:16.613 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:16.761 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (149 ms) 2024-08-16T02:26:16.761 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-08-16T02:26:16.761 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:16.945 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (184 ms) 2024-08-16T02:26:16.945 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-08-16T02:26:16.945 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:17.077 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:17.112 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (167 ms) 2024-08-16T02:26:17.112 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-08-16T02:26:17.112 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:17.269 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (157 ms) 2024-08-16T02:26:17.269 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-08-16T02:26:17.269 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:17.392 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:17.423 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (154 ms) 2024-08-16T02:26:17.425 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-08-16T02:26:17.425 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:17.601 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (178 ms) 2024-08-16T02:26:17.602 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-08-16T02:26:17.602 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:17.742 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:17.780 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (180 ms) 2024-08-16T02:26:17.781 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-08-16T02:26:17.781 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:17.968 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (187 ms) 2024-08-16T02:26:17.968 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-08-16T02:26:17.968 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:18.140 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:18.183 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (215 ms) 2024-08-16T02:26:18.183 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-08-16T02:26:18.183 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:18.599 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (416 ms) 2024-08-16T02:26:18.599 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-08-16T02:26:18.599 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:18.863 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:19.252 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (653 ms) 2024-08-16T02:26:19.252 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-08-16T02:26:19.252 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:19.717 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (464 ms) 2024-08-16T02:26:19.717 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-08-16T02:26:19.717 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:20.093 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:20.512 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (796 ms) 2024-08-16T02:26:20.512 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-08-16T02:26:20.512 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:25.323 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (4811 ms) 2024-08-16T02:26:25.324 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-08-16T02:26:25.324 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:29.708 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:30.065 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (4742 ms) 2024-08-16T02:26:30.065 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-08-16T02:26:30.065 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:26:30.269 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:26:30.269 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:26:30.269 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:26:30.269 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:26:30.269 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:26:30.269 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.270 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:26:30.270 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:30.285 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:30.285 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:26:30.285 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.285 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:26:30.285 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:30.300 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:30.300 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:26:30.301 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.301 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:26:30.301 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:30.317 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:30.317 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:26:30.317 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.317 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:26:30.317 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:30.335 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:30.335 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:26:30.335 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.335 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:26:30.335 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:30.356 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:30.356 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:26:30.357 INFO:tasks.workunit.client.0.smithi067.stdout:read: 512 2024-08-16T02:26:30.357 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.357 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:26:30.357 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:26:30.362 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:26:30.363 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:26:30.363 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.363 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:26:30.363 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.368 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:26:30.369 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:26:30.374 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:26:30.374 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:26:30.374 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.374 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:26:30.374 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:26:30.379 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:26:30.379 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:26:30.379 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:30.379 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:26:30.379 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:26:51.047 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:26:51.047 INFO:tasks.workunit.client.0.smithi067.stdout:finished read 2024-08-16T02:26:51.047 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:26:51.047 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.047 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.047 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 0 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 16384 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.048 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 49152 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 65536 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:26:51.049 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 98304 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 114688 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.050 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 147456 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 163840 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.051 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 196608 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 212992 2024-08-16T02:26:51.052 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:51.053 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:51.063 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:51.063 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:51.063 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.064 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.064 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:51.064 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:51.064 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:51.272 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:51.272 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:51.272 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.272 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.272 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:51.273 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:51.273 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:51.402 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:51.403 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:51.403 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:51.951 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:51.952 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:51.952 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:51.952 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:51.952 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:51.952 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:51.952 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:52.405 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:52.405 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:52.406 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:52.406 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:52.406 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:52.406 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:52.406 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:53.164 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:53.164 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:53.165 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:53.165 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:53.165 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:53.165 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:53.165 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:53.673 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:53.673 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:53.673 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:53.674 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:53.675 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:54.776 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:54.777 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:54.777 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:54.777 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:54.777 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:54.777 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:54.777 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:55.551 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:55.552 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:55.552 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:55.552 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:55.552 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:55.552 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:55.552 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:57.065 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:57.065 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:57.065 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:57.065 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:57.065 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:57.065 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:57.065 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:57.893 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:57.894 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:57.894 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:57.894 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:57.894 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:57.894 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:57.894 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:57.894 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:57.894 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:57.895 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:26:57.895 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:57.895 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:57.895 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:26:59.520 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:26:59.521 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:26:59.521 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:26:59.521 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:26:59.521 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:26:59.521 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:26:59.521 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:00.690 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:00.690 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:27:00.690 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:27:00.690 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:27:00.690 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:00.690 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:00.690 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:02.522 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:02.523 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:27:02.523 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:27:02.523 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:27:02.523 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:02.523 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:02.523 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:03.855 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:03.856 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:06.158 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:06.158 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:27:06.158 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:27:06.158 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:27:06.158 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:06.159 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:06.159 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:07.565 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:07.565 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:27:07.565 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:27:07.565 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:27:07.565 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:07.565 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:07.565 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:10.049 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:10.049 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:27:10.049 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:27:10.050 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:27:10.050 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:10.050 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:10.050 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:11.678 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:11.678 INFO:tasks.workunit.client.0.smithi067.stdout:finished writesame 2024-08-16T02:27:11.678 INFO:tasks.workunit.client.0.smithi067.stdout:to verify the data 2024-08-16T02:27:11.678 INFO:tasks.workunit.client.0.smithi067.stdout:verified 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:started writesame 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:expected fail, finished writesame 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (41613 ms) 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-08-16T02:27:11.679 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 0 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 0 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 1 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:snap: orig 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:found orig with size 2097152 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:read: 80 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:write test data! 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 80 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 2 2024-08-16T02:27:13.103 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 2 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:snap: orig 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:snap: written 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:found orig with size 2097152 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:found written with size 2097152 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:read: 80 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:read: 80 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:read: 80 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:write to snapshot returned -30 2024-08-16T02:27:13.104 INFO:tasks.workunit.client.0.smithi067.stdout:Read-only file system 2024-08-16T02:27:13.254 INFO:tasks.workunit.client.0.smithi067.stdout:read: 80 2024-08-16T02:27:13.255 INFO:tasks.workunit.client.0.smithi067.stdout:read: 80 2024-08-16T02:27:13.255 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 80 2024-08-16T02:27:13.255 INFO:tasks.workunit.client.0.smithi067.stdout:opening testimg@orig 2024-08-16T02:27:13.255 INFO:tasks.workunit.client.0.smithi067.stdout:read: 80 2024-08-16T02:27:13.255 INFO:tasks.workunit.client.0.smithi067.stdout:write to snapshot returned -30 2024-08-16T02:27:13.255 INFO:tasks.workunit.client.0.smithi067.stdout:Read-only file system 2024-08-16T02:27:15.124 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 2 2024-08-16T02:27:15.124 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 2 2024-08-16T02:27:15.124 INFO:tasks.workunit.client.0.smithi067.stdout:snap: orig 2024-08-16T02:27:15.124 INFO:tasks.workunit.client.0.smithi067.stdout:snap: written 2024-08-16T02:27:15.124 INFO:tasks.workunit.client.0.smithi067.stdout:found orig with size 2097152 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:found written with size 2097152 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 1 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 1 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:snap: orig 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:found orig with size 2097152 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:num snaps is: 0 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:expected: 0 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (3446 ms) 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-08-16T02:27:15.125 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (1940 ms) 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestClone 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:6444: Skipped 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestClone (0 ms) 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestClone2 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:6608: Skipped 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:17.065 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestClone2 (0 ms) 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ListChildren 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:6841: Skipped 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ListChildren (0 ms) 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:6995: Skipped 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered (0 ms) 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.LockingPP 2024-08-16T02:27:17.066 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:17.605 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.LockingPP (541 ms) 2024-08-16T02:27:17.605 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.FlushAio 2024-08-16T02:27:17.606 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:21.452 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.FlushAio (3847 ms) 2024-08-16T02:27:21.452 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-08-16T02:27:21.452 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:25.072 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.FlushAioPP (3620 ms) 2024-08-16T02:27:25.073 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-08-16T02:27:25.073 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:25.281 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (209 ms) 2024-08-16T02:27:25.281 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-08-16T02:27:25.281 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:25.437 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (156 ms) 2024-08-16T02:27:25.437 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-08-16T02:27:25.437 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:25.623 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.ZeroLengthRead (186 ms) 2024-08-16T02:27:25.623 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-08-16T02:27:25.623 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:25.892 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.LargeCacheRead (268 ms) 2024-08-16T02:27:25.892 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-08-16T02:27:25.892 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9152: Skipped 2024-08-16T02:27:25.892 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:25.892 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestPendingAio (0 ms) 2024-08-16T02:27:25.893 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-08-16T02:27:25.893 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9264: Skipped 2024-08-16T02:27:25.893 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:25.893 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyup (0 ms) 2024-08-16T02:27:25.893 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-08-16T02:27:25.894 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9325: Skipped 2024-08-16T02:27:25.894 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:25.894 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyupMismatch (0 ms) 2024-08-16T02:27:25.894 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.Flatten 2024-08-16T02:27:25.894 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9336: Skipped 2024-08-16T02:27:25.895 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:25.895 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.Flatten (0 ms) 2024-08-16T02:27:25.895 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.Sparsify 2024-08-16T02:27:25.895 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:26.091 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 32768 2024-08-16T02:27:26.091 INFO:tasks.workunit.client.0.smithi067.stdout:read: 32768 2024-08-16T02:27:26.091 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.Sparsify (199 ms) 2024-08-16T02:27:26.091 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-08-16T02:27:26.091 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:26.304 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.SparsifyPP (214 ms) 2024-08-16T02:27:26.304 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-08-16T02:27:26.304 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:31.115 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.SnapshotLimit (4811 ms) 2024-08-16T02:27:31.116 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-08-16T02:27:31.116 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:36.117 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5000 ms) 2024-08-16T02:27:36.117 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-08-16T02:27:36.117 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9523: Skipped 2024-08-16T02:27:36.117 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.117 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RebuildObjectMapViaLockOwner (0 ms) 2024-08-16T02:27:36.117 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9575: Skipped 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RenameViaLockOwner (0 ms) 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9615: Skipped 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapCreateViaLockOwner (0 ms) 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9659: Skipped 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveViaLockOwner (0 ms) 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-08-16T02:27:36.118 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9699: Skipped 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.UpdateFeaturesViaLockOwner (0 ms) 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9736: Skipped 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9777: Skipped 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapRemove2 (0 ms) 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9823: Skipped 2024-08-16T02:27:36.119 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapRenameViaLockOwner (0 ms) 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9864: Skipped 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapProtectViaLockOwner (0 ms) 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9905: Skipped 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapUnprotectViaLockOwner (0 ms) 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9949: Skipped 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.120 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.FlattenViaLockOwner (0 ms) 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:9997: Skipped 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ResizeViaLockOwner (0 ms) 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10033: Skipped 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SparsifyViaLockOwner (0 ms) 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10069: Skipped 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ObjectMapConsistentSnap (0 ms) 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.Metadata 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10121: Skipped 2024-08-16T02:27:36.121 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.Metadata (0 ms) 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10310: Skipped 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.MetadataPP (0 ms) 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10391: Skipped 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.UpdateFeatures (0 ms) 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-08-16T02:27:36.122 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:36.253 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.RebuildObjectMap (138 ms) 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10584: Skipped 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RebuildNewObjectMap (0 ms) 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10609: Skipped 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CheckObjectMap (0 ms) 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-08-16T02:27:36.254 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:36.421 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.BlockingAIO (168 ms) 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10737: Skipped 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockTransition (0 ms) 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10793: Skipped 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:10846: Skipped 2024-08-16T02:27:36.422 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:36.423 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CacheMayCopyOnWrite (0 ms) 2024-08-16T02:27:36.423 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-08-16T02:27:36.423 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:37.177 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (756 ms) 2024-08-16T02:27:37.178 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-08-16T02:27:42.031 INFO:tasks.workunit.client.0.smithi067.stdout:0.00% done 2024-08-16T02:27:42.031 INFO:tasks.workunit.client.0.smithi067.stdout:100.00% done 2024-08-16T02:27:42.031 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestImageOptions (4854 ms) 2024-08-16T02:27:42.031 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-08-16T02:27:44.500 INFO:tasks.workunit.client.0.smithi067.stdout:0.00% done 2024-08-16T02:27:44.501 INFO:tasks.workunit.client.0.smithi067.stdout:100.00% done 2024-08-16T02:27:44.501 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (2469 ms) 2024-08-16T02:27:44.501 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-08-16T02:27:44.501 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2024-08-16T02:27:44.501 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-08-16T02:27:44.501 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-08-16T02:27:44.501 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-08-16T02:27:44.501 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:44.854 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:27:44.854 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:27:44.854 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:27:44.854 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:wrote: 512 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:44.855 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:44.856 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:27:44.856 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.856 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:27:44.856 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:44.856 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:44.856 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:27:44.856 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:started write 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:write completion cb called! 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 0 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:finished write 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:27:44.857 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:27:44.858 INFO:tasks.workunit.client.0.smithi067.stdout:created completion 2024-08-16T02:27:44.859 INFO:tasks.workunit.client.0.smithi067.stdout:started read 2024-08-16T02:27:44.859 INFO:tasks.workunit.client.0.smithi067.stdout:read completion cb called! 2024-08-16T02:27:44.859 INFO:tasks.workunit.client.0.smithi067.stdout:return value is: 512 2024-08-16T02:27:44.859 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.ImagePollIO (354 ms) 2024-08-16T02:27:44.859 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.Mirror 2024-08-16T02:27:44.915 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:44.963 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:45.364 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.Mirror (510 ms) 2024-08-16T02:27:45.365 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-08-16T02:27:45.506 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (142 ms) 2024-08-16T02:27:45.506 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11322: Skipped 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CreateWithMirrorEnabled (0 ms) 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11364: Skipped 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (0 ms) 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11410: Skipped 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:45.507 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLock (0 ms) 2024-08-16T02:27:45.508 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.BreakLock 2024-08-16T02:27:45.508 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11553: Skipped 2024-08-16T02:27:45.508 INFO:tasks.workunit.client.0.smithi067.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:27:45.508 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.BreakLock (0 ms) 2024-08-16T02:27:45.508 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-08-16T02:27:45.508 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:45.613 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (106 ms) 2024-08-16T02:27:45.613 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-08-16T02:27:45.613 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-08-16T02:27:45.613 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-08-16T02:27:45.613 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11676: Skipped 2024-08-16T02:27:45.613 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:45.613 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurge (0 ms) 2024-08-16T02:27:45.613 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11722: Skipped 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (0 ms) 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11753: Skipped 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestTrashPurge (0 ms) 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:11810: Skipped 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:45.614 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndRestore (0 ms) 2024-08-16T02:27:45.615 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-08-16T02:27:45.615 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:45.836 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestListWatchers (224 ms) 2024-08-16T02:27:45.836 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-08-16T02:27:45.836 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:46.664 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.TestSetSnapById (828 ms) 2024-08-16T02:27:46.664 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.Namespaces 2024-08-16T02:27:46.988 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.Namespaces (325 ms) 2024-08-16T02:27:46.989 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-08-16T02:27:47.406 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:47.420 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.NamespacesPP (431 ms) 2024-08-16T02:27:47.420 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.Migration 2024-08-16T02:27:47.421 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:47.421 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:52.044 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.Migration (4623 ms) 2024-08-16T02:27:52.044 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-08-16T02:27:52.044 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:52.044 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:27:56.528 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.MigrationPP (4484 ms) 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12177: Skipped 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestGetAccessTimestamp (0 ms) 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12202: Skipped 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestGetModifyTimestamp (0 ms) 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12225: Skipped 2024-08-16T02:27:56.529 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ZeroOverlapFlatten (0 ms) 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12258: Skipped 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.PoolMetadata (0 ms) 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12370: Skipped 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.PoolMetadataPP (0 ms) 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.Config 2024-08-16T02:27:56.530 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12422: Skipped 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.Config (0 ms) 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12506: Skipped 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ConfigPP (0 ms) 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12583: Skipped 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.PoolStatsPP (0 ms) 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-08-16T02:27:56.531 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12658: Skipped 2024-08-16T02:27:56.532 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.532 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ImageSpec (0 ms) 2024-08-16T02:27:56.532 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-08-16T02:27:56.532 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:12853: Skipped 2024-08-16T02:27:56.532 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:27:56.532 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveWithChildMissing (0 ms) 2024-08-16T02:27:56.532 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-08-16T02:27:56.532 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:28:02.134 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.QuiesceWatch (5607 ms) 2024-08-16T02:28:02.134 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-08-16T02:28:02.134 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:28:08.223 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (6088 ms) 2024-08-16T02:28:08.223 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-08-16T02:28:08.223 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:13115: Skipped 2024-08-16T02:28:08.223 INFO:tasks.workunit.client.0.smithi067.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:28:08.223 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError (0 ms) 2024-08-16T02:28:08.223 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-08-16T02:28:08.223 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:28:08.332 INFO:tasks.workunit.client.0.smithi067.stderr:test quiesce is not long enough to time out 2024-08-16T02:28:16.990 INFO:tasks.workunit.client.0.smithi067.stderr:test quiesce is timed out 2024-08-16T02:28:17.014 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 0 2024-08-16T02:28:18.014 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 1 2024-08-16T02:28:19.014 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 2 2024-08-16T02:28:20.014 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 3 2024-08-16T02:28:21.015 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 4 2024-08-16T02:28:22.015 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 5 2024-08-16T02:28:23.015 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 6 2024-08-16T02:28:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 7 2024-08-16T02:28:25.015 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 8 2024-08-16T02:28:26.015 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 9 2024-08-16T02:28:27.015 INFO:tasks.workunit.client.0.smithi067.stderr:waiting for timed out ... 10 2024-08-16T02:28:38.016 INFO:tasks.workunit.client.0.smithi067.stderr:test retry succeeds 2024-08-16T02:28:41.158 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (32936 ms) 2024-08-16T02:28:41.158 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-08-16T02:28:41.158 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:28:41.400 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.WriteZeroes (242 ms) 2024-08-16T02:28:41.400 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-08-16T02:28:41.401 INFO:tasks.workunit.client.0.smithi067.stdout:using old format 2024-08-16T02:28:41.719 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (319 ms) 2024-08-16T02:28:41.720 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.ConcurrentOperations 2024-08-16T02:28:41.720 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:13474: Skipped 2024-08-16T02:28:41.720 INFO:tasks.workunit.client.0.smithi067.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:28:41.720 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations (0 ms) 2024-08-16T02:28:41.720 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] TestLibRBD.FormatAndCloneFormatOptions 2024-08-16T02:28:41.720 INFO:tasks.workunit.client.0.smithi067.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librbd/test_librbd.cc:13630: Skipped 2024-08-16T02:28:41.720 INFO:tasks.workunit.client.0.smithi067.stdout:Skipping due to unmet REQUIRE 2024-08-16T02:28:41.720 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.FormatAndCloneFormatOptions (0 ms) 2024-08-16T02:28:47.787 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 171 tests from TestLibRBD (500536 ms total) 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment tear-down 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] 171 tests from 1 test suite ran. (506655 ms total) 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ PASSED ] 103 tests. 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] 67 tests, listed below: 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp 2024-08-16T02:28:47.788 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespace 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestClone 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestClone2 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ListChildren 2024-08-16T02:28:47.789 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestPendingAio 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyup 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.Flatten 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RenameViaLockOwner 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapCreateViaLockOwner 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveViaLockOwner 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapRemove2 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapRenameViaLockOwner 2024-08-16T02:28:47.790 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapProtectViaLockOwner 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapUnprotectViaLockOwner 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.FlattenViaLockOwner 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ResizeViaLockOwner 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SparsifyViaLockOwner 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ObjectMapConsistentSnap 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.Metadata 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.MetadataPP 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.UpdateFeatures 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.RebuildNewObjectMap 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CheckObjectMap 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockTransition 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CacheMayCopyOnWrite 2024-08-16T02:28:47.791 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.CreateWithMirrorEnabled 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLock 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.BreakLock 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurge 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestTrashPurge 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndRestore 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestGetAccessTimestamp 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.TestGetModifyTimestamp 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ZeroOverlapFlatten 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.PoolMetadata 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.PoolMetadataPP 2024-08-16T02:28:47.792 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.Config 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ConfigPP 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.PoolStatsPP 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ImageSpec 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveWithChildMissing 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ SKIPPED ] TestLibRBD.FormatAndCloneFormatOptions 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ FAILED ] 1 test, listed below: 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout: 1 FAILED TEST 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout: YOU HAVE 1 DISABLED TEST 2024-08-16T02:28:47.793 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-08-16T02:28:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:28:47.798 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2024-08-16T02:28:47.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-16T02:28:48.283 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi067 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-08-16T02:28:48.468 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2d0bcf10a075461cb543b68553df0467 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi067 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-08-16T02:28:48.471 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-16T02:28:48.480 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-16T02:28:48.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:48.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:48.667+0000 7fe3e6aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:48.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:48.678+0000 7fe3e6aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:48.913 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:28:48.914 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-08-16T02:28:48.939 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-08-16T02:28:47.781504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":369572,"kb_used_data":369572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156523612,"statfs":{"total":160658620416,"available":160280178688,"internally_reserved":0,"allocated":378441728,"data_stored":378441728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168059960,"num_objects":-226,"num_object_clones":0,"num_object_copies":-452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28502,"num_read_kb":-145558,"num_write":-1589,"num_write_kb":-203336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.002775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":318,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.879938+0000","last_change":"2024-08-16T02:19:15.641320+0000","last_active":"2024-08-16T02:28:43.879938+0000","last_peered":"2024-08-16T02:28:43.879938+0000","last_clean":"2024-08-16T02:28:43.879938+0000","last_became_active":"2024-08-16T02:19:13.709659+0000","last_became_peered":"2024-08-16T02:19:13.709659+0000","last_unstale":"2024-08-16T02:28:43.879938+0000","last_undegraded":"2024-08-16T02:28:43.879938+0000","last_fullsized":"2024-08-16T02:28:43.879938+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":318,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.861554+0000","last_change":"2024-08-16T02:19:15.647014+0000","last_active":"2024-08-16T02:28:43.861554+0000","last_peered":"2024-08-16T02:28:43.861554+0000","last_clean":"2024-08-16T02:28:43.861554+0000","last_became_active":"2024-08-16T02:19:13.799083+0000","last_became_peered":"2024-08-16T02:19:13.799083+0000","last_unstale":"2024-08-16T02:28:43.861554+0000","last_undegraded":"2024-08-16T02:28:43.861554+0000","last_fullsized":"2024-08-16T02:28:43.861554+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":318,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.857257+0000","last_change":"2024-08-16T02:19:15.640509+0000","last_active":"2024-08-16T02:28:43.857257+0000","last_peered":"2024-08-16T02:28:43.857257+0000","last_clean":"2024-08-16T02:28:43.857257+0000","last_became_active":"2024-08-16T02:19:13.722578+0000","last_became_peered":"2024-08-16T02:19:13.722578+0000","last_unstale":"2024-08-16T02:28:43.857257+0000","last_undegraded":"2024-08-16T02:28:43.857257+0000","last_fullsized":"2024-08-16T02:28:43.857257+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":318,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.843236+0000","last_change":"2024-08-16T02:19:15.648434+0000","last_active":"2024-08-16T02:28:43.843236+0000","last_peered":"2024-08-16T02:28:43.843236+0000","last_clean":"2024-08-16T02:28:43.843236+0000","last_became_active":"2024-08-16T02:19:13.794874+0000","last_became_peered":"2024-08-16T02:19:13.794874+0000","last_unstale":"2024-08-16T02:28:43.843236+0000","last_undegraded":"2024-08-16T02:28:43.843236+0000","last_fullsized":"2024-08-16T02:28:43.843236+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":329,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:46.886582+0000","last_change":"2024-08-16T02:19:15.656601+0000","last_active":"2024-08-16T02:28:46.886582+0000","last_peered":"2024-08-16T02:28:46.886582+0000","last_clean":"2024-08-16T02:28:46.886582+0000","last_became_active":"2024-08-16T02:19:13.807895+0000","last_became_peered":"2024-08-16T02:19:13.807895+0000","last_unstale":"2024-08-16T02:28:46.886582+0000","last_undegraded":"2024-08-16T02:28:46.886582+0000","last_fullsized":"2024-08-16T02:28:46.886582+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":321,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-08-16T02:28:44.873914+0000","last_change":"2024-08-16T02:19:15.653108+0000","last_active":"2024-08-16T02:28:44.873914+0000","last_peered":"2024-08-16T02:28:44.873914+0000","last_clean":"2024-08-16T02:28:44.873914+0000","last_became_active":"2024-08-16T02:19:13.747889+0000","last_became_peered":"2024-08-16T02:19:13.747889+0000","last_unstale":"2024-08-16T02:28:44.873914+0000","last_undegraded":"2024-08-16T02:28:44.873914+0000","last_fullsized":"2024-08-16T02:28:44.873914+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":321,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-08-16T02:28:44.904622+0000","last_change":"2024-08-16T02:19:15.647826+0000","last_active":"2024-08-16T02:28:44.904622+0000","last_peered":"2024-08-16T02:28:44.904622+0000","last_clean":"2024-08-16T02:28:44.904622+0000","last_became_active":"2024-08-16T02:19:13.820022+0000","last_became_peered":"2024-08-16T02:19:13.820022+0000","last_unstale":"2024-08-16T02:28:44.904622+0000","last_undegraded":"2024-08-16T02:28:44.904622+0000","last_fullsized":"2024-08-16T02:28:44.904622+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":319,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.821646+0000","last_change":"2024-08-16T02:19:15.648266+0000","last_active":"2024-08-16T02:28:43.821646+0000","last_peered":"2024-08-16T02:28:43.821646+0000","last_clean":"2024-08-16T02:28:43.821646+0000","last_became_active":"2024-08-16T02:19:13.719057+0000","last_became_peered":"2024-08-16T02:19:13.719057+0000","last_unstale":"2024-08-16T02:28:43.821646+0000","last_undegraded":"2024-08-16T02:28:43.821646+0000","last_fullsized":"2024-08-16T02:28:43.821646+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":387,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.838073+0000","last_change":"2024-08-16T02:19:12.699715+0000","last_active":"2024-08-16T02:28:43.838073+0000","last_peered":"2024-08-16T02:28:43.838073+0000","last_clean":"2024-08-16T02:28:43.838073+0000","last_became_active":"2024-08-16T02:19:12.660251+0000","last_became_peered":"2024-08-16T02:19:12.660251+0000","last_unstale":"2024-08-16T02:28:43.838073+0000","last_undegraded":"2024-08-16T02:28:43.838073+0000","last_fullsized":"2024-08-16T02:28:43.838073+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_clean_scrub_stamp":"2024-08-16T02:19:11.588524+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160283553792,"internally_reserved":0,"allocated":375066624,"data_stored":375066624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53401182208,"internally_reserved":0,"allocated":151691264,"data_stored":151691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":80780,"kb_used_data":80780,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52216948,"statfs":{"total":53552873472,"available":53470154752,"internally_reserved":0,"allocated":82718720,"data_stored":82718720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":148136,"kb_used_data":148136,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52149592,"statfs":{"total":53552873472,"available":53401182208,"internally_reserved":0,"allocated":151691264,"data_stored":151691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":140656,"kb_used_data":140656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52157072,"statfs":{"total":53552873472,"available":53408841728,"internally_reserved":0,"allocated":144031744,"data_stored":144031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53401182208,"internally_reserved":0,"allocated":151691264,"data_stored":151691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53410189312,"internally_reserved":0,"allocated":142684160,"data_stored":142684160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53401182208,"internally_reserved":0,"allocated":151691264,"data_stored":151691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53472182272,"internally_reserved":0,"allocated":80691200,"data_stored":80691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:48.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:49.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:49.126+0000 7fe4ef952640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:49.137+0000 7fe4ef952640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:28:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-08-16T02:28:49.388 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-08-16T02:28:47.781504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":369572,"kb_used_data":369572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156523612,"statfs":{"total":160658620416,"available":160280178688,"internally_reserved":0,"allocated":378441728,"data_stored":378441728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168059960,"num_objects":-226,"num_object_clones":0,"num_object_copies":-452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28502,"num_read_kb":-145558,"num_write":-1589,"num_write_kb":-203336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-78,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.002775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":318,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.879938+0000","last_change":"2024-08-16T02:19:15.641320+0000","last_active":"2024-08-16T02:28:43.879938+0000","last_peered":"2024-08-16T02:28:43.879938+0000","last_clean":"2024-08-16T02:28:43.879938+0000","last_became_active":"2024-08-16T02:19:13.709659+0000","last_became_peered":"2024-08-16T02:19:13.709659+0000","last_unstale":"2024-08-16T02:28:43.879938+0000","last_undegraded":"2024-08-16T02:28:43.879938+0000","last_fullsized":"2024-08-16T02:28:43.879938+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":318,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.861554+0000","last_change":"2024-08-16T02:19:15.647014+0000","last_active":"2024-08-16T02:28:43.861554+0000","last_peered":"2024-08-16T02:28:43.861554+0000","last_clean":"2024-08-16T02:28:43.861554+0000","last_became_active":"2024-08-16T02:19:13.799083+0000","last_became_peered":"2024-08-16T02:19:13.799083+0000","last_unstale":"2024-08-16T02:28:43.861554+0000","last_undegraded":"2024-08-16T02:28:43.861554+0000","last_fullsized":"2024-08-16T02:28:43.861554+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":318,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.857257+0000","last_change":"2024-08-16T02:19:15.640509+0000","last_active":"2024-08-16T02:28:43.857257+0000","last_peered":"2024-08-16T02:28:43.857257+0000","last_clean":"2024-08-16T02:28:43.857257+0000","last_became_active":"2024-08-16T02:19:13.722578+0000","last_became_peered":"2024-08-16T02:19:13.722578+0000","last_unstale":"2024-08-16T02:28:43.857257+0000","last_undegraded":"2024-08-16T02:28:43.857257+0000","last_fullsized":"2024-08-16T02:28:43.857257+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":318,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.843236+0000","last_change":"2024-08-16T02:19:15.648434+0000","last_active":"2024-08-16T02:28:43.843236+0000","last_peered":"2024-08-16T02:28:43.843236+0000","last_clean":"2024-08-16T02:28:43.843236+0000","last_became_active":"2024-08-16T02:19:13.794874+0000","last_became_peered":"2024-08-16T02:19:13.794874+0000","last_unstale":"2024-08-16T02:28:43.843236+0000","last_undegraded":"2024-08-16T02:28:43.843236+0000","last_fullsized":"2024-08-16T02:28:43.843236+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":329,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:46.886582+0000","last_change":"2024-08-16T02:19:15.656601+0000","last_active":"2024-08-16T02:28:46.886582+0000","last_peered":"2024-08-16T02:28:46.886582+0000","last_clean":"2024-08-16T02:28:46.886582+0000","last_became_active":"2024-08-16T02:19:13.807895+0000","last_became_peered":"2024-08-16T02:19:13.807895+0000","last_unstale":"2024-08-16T02:28:46.886582+0000","last_undegraded":"2024-08-16T02:28:46.886582+0000","last_fullsized":"2024-08-16T02:28:46.886582+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":321,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-08-16T02:28:44.873914+0000","last_change":"2024-08-16T02:19:15.653108+0000","last_active":"2024-08-16T02:28:44.873914+0000","last_peered":"2024-08-16T02:28:44.873914+0000","last_clean":"2024-08-16T02:28:44.873914+0000","last_became_active":"2024-08-16T02:19:13.747889+0000","last_became_peered":"2024-08-16T02:19:13.747889+0000","last_unstale":"2024-08-16T02:28:44.873914+0000","last_undegraded":"2024-08-16T02:28:44.873914+0000","last_fullsized":"2024-08-16T02:28:44.873914+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":321,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-08-16T02:28:44.904622+0000","last_change":"2024-08-16T02:19:15.647826+0000","last_active":"2024-08-16T02:28:44.904622+0000","last_peered":"2024-08-16T02:28:44.904622+0000","last_clean":"2024-08-16T02:28:44.904622+0000","last_became_active":"2024-08-16T02:19:13.820022+0000","last_became_peered":"2024-08-16T02:19:13.820022+0000","last_unstale":"2024-08-16T02:28:44.904622+0000","last_undegraded":"2024-08-16T02:28:44.904622+0000","last_fullsized":"2024-08-16T02:28:44.904622+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":319,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.821646+0000","last_change":"2024-08-16T02:19:15.648266+0000","last_active":"2024-08-16T02:28:43.821646+0000","last_peered":"2024-08-16T02:28:43.821646+0000","last_clean":"2024-08-16T02:28:43.821646+0000","last_became_active":"2024-08-16T02:19:13.719057+0000","last_became_peered":"2024-08-16T02:19:13.719057+0000","last_unstale":"2024-08-16T02:28:43.821646+0000","last_undegraded":"2024-08-16T02:28:43.821646+0000","last_fullsized":"2024-08-16T02:28:43.821646+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":387,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-08-16T02:28:43.838073+0000","last_change":"2024-08-16T02:19:12.699715+0000","last_active":"2024-08-16T02:28:43.838073+0000","last_peered":"2024-08-16T02:28:43.838073+0000","last_clean":"2024-08-16T02:28:43.838073+0000","last_became_active":"2024-08-16T02:19:12.660251+0000","last_became_peered":"2024-08-16T02:19:12.660251+0000","last_unstale":"2024-08-16T02:28:43.838073+0000","last_undegraded":"2024-08-16T02:28:43.838073+0000","last_fullsized":"2024-08-16T02:28:43.838073+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_clean_scrub_stamp":"2024-08-16T02:19:11.588524+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160283553792,"internally_reserved":0,"allocated":375066624,"data_stored":375066624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53401182208,"internally_reserved":0,"allocated":151691264,"data_stored":151691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":80780,"kb_used_data":80780,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52216948,"statfs":{"total":53552873472,"available":53470154752,"internally_reserved":0,"allocated":82718720,"data_stored":82718720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738942,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":148136,"kb_used_data":148136,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52149592,"statfs":{"total":53552873472,"available":53401182208,"internally_reserved":0,"allocated":151691264,"data_stored":151691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":140656,"kb_used_data":140656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52157072,"statfs":{"total":53552873472,"available":53408841728,"internally_reserved":0,"allocated":144031744,"data_stored":144031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53401182208,"internally_reserved":0,"allocated":151691264,"data_stored":151691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53410189312,"internally_reserved":0,"allocated":142684160,"data_stored":142684160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53401182208,"internally_reserved":0,"allocated":151691264,"data_stored":151691264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53472182272,"internally_reserved":0,"allocated":80691200,"data_stored":80691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:49.389 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-16T02:28:49.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:49.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:49.510+0000 7f3fecd92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:49.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:49.520+0000 7f3fecd92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:28:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-08-16T02:28:49.765 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-08-16T02:28:49.698177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":304316,"kb_used_data":304316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156588868,"statfs":{"total":160658620416,"available":160347000832,"internally_reserved":0,"allocated":311619584,"data_stored":311619584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168064190,"num_objects":-222,"num_object_clones":0,"num_object_copies":-444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28103,"num_read_kb":-145411,"num_write":-1521,"num_write_kb":-203267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.919041"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.855755+0000","last_change":"2024-08-16T02:19:15.641320+0000","last_active":"2024-08-16T02:28:47.855755+0000","last_peered":"2024-08-16T02:28:47.855755+0000","last_clean":"2024-08-16T02:28:47.855755+0000","last_became_active":"2024-08-16T02:19:13.709659+0000","last_became_peered":"2024-08-16T02:19:13.709659+0000","last_unstale":"2024-08-16T02:28:47.855755+0000","last_undegraded":"2024-08-16T02:28:47.855755+0000","last_fullsized":"2024-08-16T02:28:47.855755+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.884607+0000","last_change":"2024-08-16T02:19:15.647014+0000","last_active":"2024-08-16T02:28:47.884607+0000","last_peered":"2024-08-16T02:28:47.884607+0000","last_clean":"2024-08-16T02:28:47.884607+0000","last_became_active":"2024-08-16T02:19:13.799083+0000","last_became_peered":"2024-08-16T02:19:13.799083+0000","last_unstale":"2024-08-16T02:28:47.884607+0000","last_undegraded":"2024-08-16T02:28:47.884607+0000","last_fullsized":"2024-08-16T02:28:47.884607+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.918742+0000","last_change":"2024-08-16T02:19:15.640509+0000","last_active":"2024-08-16T02:28:47.918742+0000","last_peered":"2024-08-16T02:28:47.918742+0000","last_clean":"2024-08-16T02:28:47.918742+0000","last_became_active":"2024-08-16T02:19:13.722578+0000","last_became_peered":"2024-08-16T02:19:13.722578+0000","last_unstale":"2024-08-16T02:28:47.918742+0000","last_undegraded":"2024-08-16T02:28:47.918742+0000","last_fullsized":"2024-08-16T02:28:47.918742+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:49.211080+0000","last_change":"2024-08-16T02:19:15.648434+0000","last_active":"2024-08-16T02:28:49.211080+0000","last_peered":"2024-08-16T02:28:49.211080+0000","last_clean":"2024-08-16T02:28:49.211080+0000","last_became_active":"2024-08-16T02:19:13.794874+0000","last_became_peered":"2024-08-16T02:19:13.794874+0000","last_unstale":"2024-08-16T02:28:49.211080+0000","last_undegraded":"2024-08-16T02:28:49.211080+0000","last_fullsized":"2024-08-16T02:28:49.211080+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":329,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:46.886582+0000","last_change":"2024-08-16T02:19:15.656601+0000","last_active":"2024-08-16T02:28:46.886582+0000","last_peered":"2024-08-16T02:28:46.886582+0000","last_clean":"2024-08-16T02:28:46.886582+0000","last_became_active":"2024-08-16T02:19:13.807895+0000","last_became_peered":"2024-08-16T02:19:13.807895+0000","last_unstale":"2024-08-16T02:28:46.886582+0000","last_undegraded":"2024-08-16T02:28:46.886582+0000","last_fullsized":"2024-08-16T02:28:46.886582+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":321,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-08-16T02:28:44.873914+0000","last_change":"2024-08-16T02:19:15.653108+0000","last_active":"2024-08-16T02:28:44.873914+0000","last_peered":"2024-08-16T02:28:44.873914+0000","last_clean":"2024-08-16T02:28:44.873914+0000","last_became_active":"2024-08-16T02:19:13.747889+0000","last_became_peered":"2024-08-16T02:19:13.747889+0000","last_unstale":"2024-08-16T02:28:44.873914+0000","last_undegraded":"2024-08-16T02:28:44.873914+0000","last_fullsized":"2024-08-16T02:28:44.873914+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":321,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-08-16T02:28:44.904622+0000","last_change":"2024-08-16T02:19:15.647826+0000","last_active":"2024-08-16T02:28:44.904622+0000","last_peered":"2024-08-16T02:28:44.904622+0000","last_clean":"2024-08-16T02:28:44.904622+0000","last_became_active":"2024-08-16T02:19:13.820022+0000","last_became_peered":"2024-08-16T02:19:13.820022+0000","last_unstale":"2024-08-16T02:28:44.904622+0000","last_undegraded":"2024-08-16T02:28:44.904622+0000","last_fullsized":"2024-08-16T02:28:44.904622+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":331,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.884175+0000","last_change":"2024-08-16T02:19:15.648266+0000","last_active":"2024-08-16T02:28:47.884175+0000","last_peered":"2024-08-16T02:28:47.884175+0000","last_clean":"2024-08-16T02:28:47.884175+0000","last_became_active":"2024-08-16T02:19:13.719057+0000","last_became_peered":"2024-08-16T02:19:13.719057+0000","last_unstale":"2024-08-16T02:28:47.884175+0000","last_undegraded":"2024-08-16T02:28:47.884175+0000","last_fullsized":"2024-08-16T02:28:47.884175+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":399,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.921988+0000","last_change":"2024-08-16T02:19:12.699715+0000","last_active":"2024-08-16T02:28:47.921988+0000","last_peered":"2024-08-16T02:28:47.921988+0000","last_clean":"2024-08-16T02:28:47.921988+0000","last_became_active":"2024-08-16T02:19:12.660251+0000","last_became_peered":"2024-08-16T02:19:12.660251+0000","last_unstale":"2024-08-16T02:28:47.921988+0000","last_undegraded":"2024-08-16T02:28:47.921988+0000","last_fullsized":"2024-08-16T02:28:47.921988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_clean_scrub_stamp":"2024-08-16T02:19:11.588524+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160350457856,"internally_reserved":0,"allocated":308162560,"data_stored":308162560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53468086272,"internally_reserved":0,"allocated":84787200,"data_stored":84787200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706234,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":80780,"kb_used_data":80780,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52216948,"statfs":{"total":53552873472,"available":53470154752,"internally_reserved":0,"allocated":82718720,"data_stored":82718720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738947,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":82880,"kb_used_data":82880,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52214848,"statfs":{"total":53552873472,"available":53468004352,"internally_reserved":0,"allocated":84869120,"data_stored":84869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":140656,"kb_used_data":140656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52157072,"statfs":{"total":53552873472,"available":53408841728,"internally_reserved":0,"allocated":144031744,"data_stored":144031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53468086272,"internally_reserved":0,"allocated":84787200,"data_stored":84787200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53410189312,"internally_reserved":0,"allocated":142684160,"data_stored":142684160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53468086272,"internally_reserved":0,"allocated":84787200,"data_stored":84787200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53472182272,"internally_reserved":0,"allocated":80691200,"data_stored":80691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:49.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:28:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:49.880+0000 7fda1362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:49.890+0000 7fda1362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:50.116 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:28:50.117 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":119,"fsid":"734e1993-fc74-4287-90f2-01825483bae0","created":"2024-08-16T02:19:02.617605+0000","modified":"2024-08-16T02:28:47.780242+0000","last_up_change":"2024-08-16T02:19:10.579561+0000","last_in_change":"2024-08-16T02:19:04.546626+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:19:11.541917+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:19:12.476276+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a550ea25-2edb-4628-ab47-f1133850bc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":4185550721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":4185550721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":4185550721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":4185550721}]},"public_addr":"172.21.15.67:6808/4185550721","cluster_addr":"172.21.15.67:6809/4185550721","heartbeat_back_addr":"172.21.15.67:6806/4185550721","heartbeat_front_addr":"172.21.15.67:6812/4185550721","state":["exists","up"]},{"osd":1,"uuid":"8b75d572-7db0-40cf-9f39-9063c0eec5de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6803","nonce":467805611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":467805611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":467805611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":467805611}]},"public_addr":"172.21.15.67:6803/467805611","cluster_addr":"172.21.15.67:6802/467805611","heartbeat_back_addr":"172.21.15.67:6805/467805611","heartbeat_front_addr":"172.21.15.67:6804/467805611","state":["exists","up"]},{"osd":2,"uuid":"e3556829-8f2e-4544-82f9-0ab185f41290","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3162681119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":3162681119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3162681119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":3162681119}]},"public_addr":"172.21.15.67:6810/3162681119","cluster_addr":"172.21.15.67:6811/3162681119","heartbeat_back_addr":"172.21.15.67:6816/3162681119","heartbeat_front_addr":"172.21.15.67:6815/3162681119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":13},{"begin":17,"length":1},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":28,"length":8}]},{"pool":4,"snaps":[{"begin":2,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:28:51.134 INFO:tasks.ceph:Scrubbing osd.0 2024-08-16T02:28:51.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:28:51.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:51.245+0000 7fe3b746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:51.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:51.257+0000 7fe3b746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:51.274 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-08-16T02:28:51.274 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "" 2024-08-16T02:28:51.274 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-08-16T02:28:51.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-16T02:28:51.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:51.394+0000 7f0c0886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:51.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:51.403+0000 7f0c0886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-08-16T02:28:51.626 INFO:tasks.ceph:Scrubbing osd.1 2024-08-16T02:28:51.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:28:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:51.736+0000 7fc28006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:51.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:51.749+0000 7fc28006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:51.764 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-08-16T02:28:51.764 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "" 2024-08-16T02:28:51.764 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-08-16T02:28:51.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-16T02:28:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:51.883+0000 7fdc0b6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:51.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:51.892+0000 7fdc0b6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:52.101 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-08-16T02:28:52.117 INFO:tasks.ceph:Scrubbing osd.2 2024-08-16T02:28:52.118 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:28:52.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:52.233+0000 7f8164213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:52.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:52.245+0000 7f8164213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:52.259 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-08-16T02:28:52.260 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "" 2024-08-16T02:28:52.260 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-08-16T02:28:52.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-16T02:28:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:52.380+0000 7f508faed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:52.389+0000 7f508faed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-08-16T02:28:52.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:28:52.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:52.730+0000 7fbb66792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:52.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:28:52.738+0000 7fbb66792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:28:52.950 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:28:52.950 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-08-16T02:28:52.962 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-08-16T02:28:51.698528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":224572,"kb_used_data":224572,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156668612,"statfs":{"total":160658620416,"available":160428658688,"internally_reserved":0,"allocated":229961728,"data_stored":229961728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168064056,"num_objects":-221,"num_object_clones":0,"num_object_copies":-442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28108,"num_read_kb":-145415,"num_write":-1523,"num_write_kb":-203268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001158"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.855755+0000","last_change":"2024-08-16T02:19:15.641320+0000","last_active":"2024-08-16T02:28:47.855755+0000","last_peered":"2024-08-16T02:28:47.855755+0000","last_clean":"2024-08-16T02:28:47.855755+0000","last_became_active":"2024-08-16T02:19:13.709659+0000","last_became_peered":"2024-08-16T02:19:13.709659+0000","last_unstale":"2024-08-16T02:28:47.855755+0000","last_undegraded":"2024-08-16T02:28:47.855755+0000","last_fullsized":"2024-08-16T02:28:47.855755+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.884607+0000","last_change":"2024-08-16T02:19:15.647014+0000","last_active":"2024-08-16T02:28:47.884607+0000","last_peered":"2024-08-16T02:28:47.884607+0000","last_clean":"2024-08-16T02:28:47.884607+0000","last_became_active":"2024-08-16T02:19:13.799083+0000","last_became_peered":"2024-08-16T02:19:13.799083+0000","last_unstale":"2024-08-16T02:28:47.884607+0000","last_undegraded":"2024-08-16T02:28:47.884607+0000","last_fullsized":"2024-08-16T02:28:47.884607+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.918742+0000","last_change":"2024-08-16T02:19:15.640509+0000","last_active":"2024-08-16T02:28:47.918742+0000","last_peered":"2024-08-16T02:28:47.918742+0000","last_clean":"2024-08-16T02:28:47.918742+0000","last_became_active":"2024-08-16T02:19:13.722578+0000","last_became_peered":"2024-08-16T02:19:13.722578+0000","last_unstale":"2024-08-16T02:28:47.918742+0000","last_undegraded":"2024-08-16T02:28:47.918742+0000","last_fullsized":"2024-08-16T02:28:47.918742+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:49.211080+0000","last_change":"2024-08-16T02:19:15.648434+0000","last_active":"2024-08-16T02:28:49.211080+0000","last_peered":"2024-08-16T02:28:49.211080+0000","last_clean":"2024-08-16T02:28:49.211080+0000","last_became_active":"2024-08-16T02:19:13.794874+0000","last_became_peered":"2024-08-16T02:19:13.794874+0000","last_unstale":"2024-08-16T02:28:49.211080+0000","last_undegraded":"2024-08-16T02:28:49.211080+0000","last_fullsized":"2024-08-16T02:28:49.211080+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":329,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-08-16T02:28:46.886582+0000","last_change":"2024-08-16T02:19:15.656601+0000","last_active":"2024-08-16T02:28:46.886582+0000","last_peered":"2024-08-16T02:28:46.886582+0000","last_clean":"2024-08-16T02:28:46.886582+0000","last_became_active":"2024-08-16T02:19:13.807895+0000","last_became_peered":"2024-08-16T02:19:13.807895+0000","last_unstale":"2024-08-16T02:28:46.886582+0000","last_undegraded":"2024-08-16T02:28:46.886582+0000","last_fullsized":"2024-08-16T02:28:46.886582+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.968949+0000","last_change":"2024-08-16T02:19:15.653108+0000","last_active":"2024-08-16T02:28:47.968949+0000","last_peered":"2024-08-16T02:28:47.968949+0000","last_clean":"2024-08-16T02:28:47.968949+0000","last_became_active":"2024-08-16T02:19:13.747889+0000","last_became_peered":"2024-08-16T02:19:13.747889+0000","last_unstale":"2024-08-16T02:28:47.968949+0000","last_undegraded":"2024-08-16T02:28:47.968949+0000","last_fullsized":"2024-08-16T02:28:47.968949+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":330,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.917386+0000","last_change":"2024-08-16T02:19:15.647826+0000","last_active":"2024-08-16T02:28:47.917386+0000","last_peered":"2024-08-16T02:28:47.917386+0000","last_clean":"2024-08-16T02:28:47.917386+0000","last_became_active":"2024-08-16T02:19:13.820022+0000","last_became_peered":"2024-08-16T02:19:13.820022+0000","last_unstale":"2024-08-16T02:28:47.917386+0000","last_undegraded":"2024-08-16T02:28:47.917386+0000","last_fullsized":"2024-08-16T02:28:47.917386+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":331,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.884175+0000","last_change":"2024-08-16T02:19:15.648266+0000","last_active":"2024-08-16T02:28:47.884175+0000","last_peered":"2024-08-16T02:28:47.884175+0000","last_clean":"2024-08-16T02:28:47.884175+0000","last_became_active":"2024-08-16T02:19:13.719057+0000","last_became_peered":"2024-08-16T02:19:13.719057+0000","last_unstale":"2024-08-16T02:28:47.884175+0000","last_undegraded":"2024-08-16T02:28:47.884175+0000","last_fullsized":"2024-08-16T02:28:47.884175+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:12.593117+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":399,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:47.921988+0000","last_change":"2024-08-16T02:19:12.699715+0000","last_active":"2024-08-16T02:28:47.921988+0000","last_peered":"2024-08-16T02:28:47.921988+0000","last_clean":"2024-08-16T02:28:47.921988+0000","last_became_active":"2024-08-16T02:19:12.660251+0000","last_became_peered":"2024-08-16T02:19:12.660251+0000","last_unstale":"2024-08-16T02:28:47.921988+0000","last_undegraded":"2024-08-16T02:28:47.921988+0000","last_fullsized":"2024-08-16T02:28:47.921988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:19:11.588524+0000","last_clean_scrub_stamp":"2024-08-16T02:19:11.588524+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160430088192,"internally_reserved":0,"allocated":228532224,"data_stored":228532224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53468086272,"internally_reserved":0,"allocated":84787200,"data_stored":84787200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1036,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296692,"statfs":{"total":53552873472,"available":53551812608,"internally_reserved":0,"allocated":1060864,"data_stored":1060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738947,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":82880,"kb_used_data":82880,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52214848,"statfs":{"total":53552873472,"available":53468004352,"internally_reserved":0,"allocated":84869120,"data_stored":84869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706236,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":140656,"kb_used_data":140656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52157072,"statfs":{"total":53552873472,"available":53408841728,"internally_reserved":0,"allocated":144031744,"data_stored":144031744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53468086272,"internally_reserved":0,"allocated":84787200,"data_stored":84787200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53410189312,"internally_reserved":0,"allocated":142684160,"data_stored":142684160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53468086272,"internally_reserved":0,"allocated":84787200,"data_stored":84787200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551812608,"internally_reserved":0,"allocated":1060864,"data_stored":1060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:28:52.965 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-16T02:19:12.593117+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=12, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.966 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-16T02:19:12.593117+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=12, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.966 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-16T02:19:12.593117+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=12, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.966 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-16T02:19:12.593117+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=12, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.966 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-16T02:19:12.593117+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=12, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.966 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-16T02:19:12.593117+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=12, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.967 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-16T02:19:12.593117+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=12, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.967 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-16T02:19:12.593117+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=12, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.967 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-16T02:19:11.588524+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=19, tm_sec=11, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=28, tm_sec=50, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:28:52.967 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-16T02:29:12.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:29:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:29:13.092+0000 7f9423264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:29:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:29:13.101+0000 7f9423264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:29:13.306 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-08-16T02:29:13.307 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-08-16T02:29:13.319 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-08-16T02:29:11.703473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5168,"kb_used_data":5168,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888016,"statfs":{"total":160658620416,"available":160653328384,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003147"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:52.107863+0000","last_change":"2024-08-16T02:28:52.138858+0000","last_active":"2024-08-16T02:28:52.107863+0000","last_peered":"2024-08-16T02:28:52.107863+0000","last_clean":"2024-08-16T02:28:52.107863+0000","last_became_active":"2024-08-16T02:19:13.709659+0000","last_became_peered":"2024-08-16T02:19:13.709659+0000","last_unstale":"2024-08-16T02:28:52.107863+0000","last_undegraded":"2024-08-16T02:28:52.107863+0000","last_fullsized":"2024-08-16T02:28:52.107863+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:52.138682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:52.138682+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:52.108427+0000","last_change":"2024-08-16T02:28:52.147837+0000","last_active":"2024-08-16T02:28:52.108427+0000","last_peered":"2024-08-16T02:28:52.108427+0000","last_clean":"2024-08-16T02:28:52.108427+0000","last_became_active":"2024-08-16T02:19:13.799083+0000","last_became_peered":"2024-08-16T02:19:13.799083+0000","last_unstale":"2024-08-16T02:28:52.108427+0000","last_undegraded":"2024-08-16T02:28:52.108427+0000","last_fullsized":"2024-08-16T02:28:52.108427+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:52.147525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:52.147525+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:52.106663+0000","last_change":"2024-08-16T02:28:52.141256+0000","last_active":"2024-08-16T02:28:52.106663+0000","last_peered":"2024-08-16T02:28:52.106663+0000","last_clean":"2024-08-16T02:28:52.106663+0000","last_became_active":"2024-08-16T02:19:13.722578+0000","last_became_peered":"2024-08-16T02:19:13.722578+0000","last_unstale":"2024-08-16T02:28:52.106663+0000","last_undegraded":"2024-08-16T02:28:52.106663+0000","last_fullsized":"2024-08-16T02:28:52.106663+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:52.141080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:52.141080+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:52.107106+0000","last_change":"2024-08-16T02:28:52.145106+0000","last_active":"2024-08-16T02:28:52.107106+0000","last_peered":"2024-08-16T02:28:52.107106+0000","last_clean":"2024-08-16T02:28:52.107106+0000","last_became_active":"2024-08-16T02:19:13.794874+0000","last_became_peered":"2024-08-16T02:19:13.794874+0000","last_unstale":"2024-08-16T02:28:52.107106+0000","last_undegraded":"2024-08-16T02:28:52.107106+0000","last_fullsized":"2024-08-16T02:28:52.107106+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:52.144913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:52.144913+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":334,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:51.612419+0000","last_change":"2024-08-16T02:28:51.631557+0000","last_active":"2024-08-16T02:28:51.612419+0000","last_peered":"2024-08-16T02:28:51.612419+0000","last_clean":"2024-08-16T02:28:51.612419+0000","last_became_active":"2024-08-16T02:19:13.807895+0000","last_became_peered":"2024-08-16T02:19:13.807895+0000","last_unstale":"2024-08-16T02:28:51.612419+0000","last_undegraded":"2024-08-16T02:28:51.612419+0000","last_fullsized":"2024-08-16T02:28:51.612419+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"14'2","last_scrub_stamp":"2024-08-16T02:28:51.631372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:51.631372+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:52.607312+0000","last_change":"2024-08-16T02:28:52.621561+0000","last_active":"2024-08-16T02:28:52.607312+0000","last_peered":"2024-08-16T02:28:52.607312+0000","last_clean":"2024-08-16T02:28:52.607312+0000","last_became_active":"2024-08-16T02:19:13.747889+0000","last_became_peered":"2024-08-16T02:19:13.747889+0000","last_unstale":"2024-08-16T02:28:52.607312+0000","last_undegraded":"2024-08-16T02:28:52.607312+0000","last_fullsized":"2024-08-16T02:28:52.607312+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:52.621387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:52.621387+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":332,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:52.604685+0000","last_change":"2024-08-16T02:28:52.617337+0000","last_active":"2024-08-16T02:28:52.604685+0000","last_peered":"2024-08-16T02:28:52.604685+0000","last_clean":"2024-08-16T02:28:52.604685+0000","last_became_active":"2024-08-16T02:19:13.820022+0000","last_became_peered":"2024-08-16T02:19:13.820022+0000","last_unstale":"2024-08-16T02:28:52.604685+0000","last_undegraded":"2024-08-16T02:28:52.604685+0000","last_fullsized":"2024-08-16T02:28:52.604685+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:28:52.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:52.617157+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:52.106135+0000","last_change":"2024-08-16T02:28:52.137308+0000","last_active":"2024-08-16T02:28:52.106135+0000","last_peered":"2024-08-16T02:28:52.106135+0000","last_clean":"2024-08-16T02:28:52.106135+0000","last_became_active":"2024-08-16T02:19:13.719057+0000","last_became_peered":"2024-08-16T02:19:13.719057+0000","last_unstale":"2024-08-16T02:28:52.106135+0000","last_undegraded":"2024-08-16T02:28:52.106135+0000","last_fullsized":"2024-08-16T02:28:52.106135+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'1","last_scrub_stamp":"2024-08-16T02:28:52.137113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:52.137113+0000","last_clean_scrub_stamp":"2024-08-16T02:19:12.593117+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":401,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-08-16T02:28:52.103964+0000","last_change":"2024-08-16T02:28:52.165499+0000","last_active":"2024-08-16T02:28:52.103964+0000","last_peered":"2024-08-16T02:28:52.103964+0000","last_clean":"2024-08-16T02:28:52.103964+0000","last_became_active":"2024-08-16T02:19:12.660251+0000","last_became_peered":"2024-08-16T02:19:12.660251+0000","last_unstale":"2024-08-16T02:28:52.103964+0000","last_undegraded":"2024-08-16T02:28:52.103964+0000","last_fullsized":"2024-08-16T02:28:52.103964+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-08-16T02:28:52.165288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:28:52.165288+0000","last_clean_scrub_stamp":"2024-08-16T02:19:11.588524+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653328384,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550628864,"internally_reserved":0,"allocated":2244608,"data_stored":2244608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1036,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296692,"statfs":{"total":53552873472,"available":53551812608,"internally_reserved":0,"allocated":1060864,"data_stored":1060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738967,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2192,"kb_used_data":2192,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295536,"statfs":{"total":53552873472,"available":53550628864,"internally_reserved":0,"allocated":2244608,"data_stored":2244608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706256,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1940,"kb_used_data":1940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295788,"statfs":{"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550628864,"internally_reserved":0,"allocated":2244608,"data_stored":2244608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550628864,"internally_reserved":0,"allocated":2244608,"data_stored":2244608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551812608,"internally_reserved":0,"allocated":1060864,"data_stored":1060864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:29:13.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-16T02:29:13.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:29:13.427+0000 7f9f1f392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:29:13.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-08-16T02:29:13.438+0000 7f9f1f392640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:29:13.666 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-16T02:29:13.667 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-16T02:29:13.667 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-16T02:29:13.667 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:29:13.759 INFO:tasks.ceph.osd.0:Stopped 2024-08-16T02:29:13.760 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-16T02:29:13.760 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:29:13.834 INFO:tasks.ceph.osd.1:Stopped 2024-08-16T02:29:13.835 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-16T02:29:13.835 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:29:13.911 INFO:tasks.ceph.osd.2:Stopped 2024-08-16T02:29:13.911 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-16T02:29:13.911 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-16T02:29:13.912 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:29:13.941 INFO:tasks.ceph.mgr.x:Stopped 2024-08-16T02:29:13.942 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-16T02:29:13.942 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-16T02:29:13.942 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:29:13.995 INFO:tasks.ceph.mon.a:Stopped 2024-08-16T02:29:13.995 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-16T02:29:13.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(POOL_FULL\)' | head -n 1 2024-08-16T02:29:14.065 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:29:14.065 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-16T02:29:14.258 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:29:14.258 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-16T02:29:14.347 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:29:14.347 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-16T02:29:14.397 INFO:tasks.ceph:Archiving mon data... 2024-08-16T02:29:14.398 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857404/data/mon.a.tgz 2024-08-16T02:29:14.398 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-08-16T02:29:14.414 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.9z82PQPiOT 2024-08-16T02:29:14.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.9z82PQPiOT 2024-08-16T02:29:14.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.9z82PQPiOT 2024-08-16T02:29:14.624 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.9z82PQPiOT is 224KB 2024-08-16T02:29:14.719 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.9z82PQPiOT 2024-08-16T02:29:14.735 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-16T02:29:14.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-16T02:29:14.857 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-16T02:29:14.857 INFO:tasks.ceph:Archiving crash dumps... 2024-08-16T02:29:14.859 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857404/remote/smithi067/crash 2024-08-16T02:29:14.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-16T02:29:14.923 INFO:tasks.ceph:Compressing logs... 2024-08-16T02:29:14.923 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-16T02:29:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34210.log 2024-08-16T02:29:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30356.log 2024-08-16T02:29:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34336.log 2024-08-16T02:29:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34210.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34210.log.gz 2024-08-16T02:29:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30764.log 2024-08-16T02:29:14.997 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30356.log.gz 2024-08-16T02:29:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33958.log 2024-08-16T02:29:14.997 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34336.log.gz 2024-08-16T02:29:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30683.log 2024-08-16T02:29:14.997 INFO:teuthology.orchestra.run.smithi067.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30764.log.gz 2024-08-16T02:29:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2024-08-16T02:29:14.998 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33958.log.gz 2024-08-16T02:29:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30683.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-16T02:29:14.998 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30683.log.gz 2024-08-16T02:29:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31167.log 2024-08-16T02:29:14.998 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2024-08-16T02:29:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-16T02:29:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34126.log 2024-08-16T02:29:14.999 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31167.log.gz 2024-08-16T02:29:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-16T02:29:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31041.log 2024-08-16T02:29:14.999 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34126.log.gz 2024-08-16T02:29:14.999 INFO:teuthology.orchestra.run.smithi067.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-16T02:29:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34042.log 2024-08-16T02:29:15.000 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30739.log 2024-08-16T02:29:15.000 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31041.log.gz 2024-08-16T02:29:15.000 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-16T02:29:15.000 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30544.log 2024-08-16T02:29:15.000 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34042.log.gz 2024-08-16T02:29:15.000 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30876.log 2024-08-16T02:29:15.000 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30739.log.gz 2024-08-16T02:29:15.000 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30067.log 2024-08-16T02:29:15.001 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30544.log.gz 2024-08-16T02:29:15.001 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34168.log 2024-08-16T02:29:15.001 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30876.log.gz 2024-08-16T02:29:15.001 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30067.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-16T02:29:15.001 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30067.log.gz 2024-08-16T02:29:15.001 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34168.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-16T02:29:15.001 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34168.log.gz 2024-08-16T02:29:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31296.log 2024-08-16T02:29:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31168.log 2024-08-16T02:29:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31296.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-16T02:29:15.014 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31296.log.gz 2024-08-16T02:29:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.31168.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30514.log 2024-08-16T02:29:15.014 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31168.log.gz 2024-08-16T02:29:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34084.log 2024-08-16T02:29:15.014 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30514.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-16T02:29:15.014 INFO:teuthology.orchestra.run.smithi067.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30514.log.gz 2024-08-16T02:29:15.015 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30276.log 2024-08-16T02:29:15.015 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34084.log.gz 2024-08-16T02:29:15.015 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29107.log 2024-08-16T02:29:15.015 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34000.log 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30276.log.gz 2024-08-16T02:29:15.015 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-08-16T02:29:15.015 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.29107.log: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29107.log.gz 2024-08-16T02:29:15.015 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31338.log 2024-08-16T02:29:15.015 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34000.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34000.log.gz 2024-08-16T02:29:15.016 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.33874.log 2024-08-16T02:29:15.016 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31338.log: gzip 50.4% -5 --verbose -- /var/log/ceph/ceph-client.admin.30918.log 2024-08-16T02:29:15.016 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31338.log.gz 2024-08-16T02:29:15.016 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33874.log: gzip -5 --verbose 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33874.log.gz 2024-08-16T02:29:15.016 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.30155.log 2024-08-16T02:29:15.016 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30918.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30918.log.gz 2024-08-16T02:29:15.016 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34252.log 2024-08-16T02:29:15.016 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30155.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30155.log.gz 2024-08-16T02:29:15.017 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34376.log 2024-08-16T02:29:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34252.log: gzip -5 50.0% --verbose -- /var/log/ceph/ceph-client.admin.31254.log 2024-08-16T02:29:15.017 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34252.log.gz 2024-08-16T02:29:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34376.log: gzip -5 --verbose 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34376.log.gz 2024-08-16T02:29:15.017 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.33916.log 2024-08-16T02:29:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31254.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31254.log.gz 2024-08-16T02:29:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31380.log 2024-08-16T02:29:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33916.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30960.log 2024-08-16T02:29:15.018 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33916.log.gz 2024-08-16T02:29:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31600.log 2024-08-16T02:29:15.018 INFO:teuthology.orchestra.run.smithi067.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31380.log.gz 2024-08-16T02:29:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34294.log 2024-08-16T02:29:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30960.log: /var/log/ceph/ceph-client.admin.31600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31040.log 2024-08-16T02:29:15.018 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30960.log.gz 2024-08-16T02:29:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34294.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31166.log 2024-08-16T02:29:15.021 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34294.log.gz 2024-08-16T02:29:15.021 INFO:teuthology.orchestra.run.smithi067.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31600.log.gz 2024-08-16T02:29:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30834.log 2024-08-16T02:29:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31040.log: /var/log/ceph/ceph-client.admin.31166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31042.log 2024-08-16T02:29:15.022 INFO:teuthology.orchestra.run.smithi067.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31040.log.gz 2024-08-16T02:29:15.022 INFO:teuthology.orchestra.run.smithi067.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31166.log.gz 2024-08-16T02:29:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30834.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30834.log.gz 2024-08-16T02:29:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31042.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31042.log.gz 2024-08-16T02:29:15.049 INFO:teuthology.orchestra.run.smithi067.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-16T02:29:15.238 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-16T02:29:16.903 INFO:teuthology.orchestra.run.smithi067.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-16T02:29:17.701 INFO:teuthology.orchestra.run.smithi067.stderr: 96.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-16T02:29:18.909 INFO:teuthology.orchestra.run.smithi067.stderr: 96.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-16T02:29:18.910 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-08-16T02:29:18.910 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m3.931s 2024-08-16T02:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m8.470s 2024-08-16T02:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.397s 2024-08-16T02:29:18.911 INFO:tasks.ceph:Archiving logs... 2024-08-16T02:29:18.911 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857404/remote/smithi067/log 2024-08-16T02:29:18.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-16T02:29:19.917 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-16T02:29:19.927 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-16T02:29:19.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-16T02:29:19.978 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-16T02:29:19.987 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-16T02:29:19.987 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:29:20.011 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:29:20.015 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:29:20.016 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-08-16T02:29:20.016 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +111us[ +106us] +/- 45ms 2024-08-16T02:29:20.016 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -683us[ -683us] +/- 65ms 2024-08-16T02:29:20.016 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +608us[ +608us] +/- 65ms 2024-08-16T02:29:20.016 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:29:20.017 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-16T02:29:20.026 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-16T02:29:20.026 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-16T02:29:20.036 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-16T02:29:20.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:29:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:29:20.148 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-16T02:29:20.157 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-16T02:29:20.165 INFO:teuthology.task.internal:Duration was 1090.765719 seconds 2024-08-16T02:29:20.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-16T02:29:20.173 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-16T02:29:20.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-16T02:29:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:29:20.547 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-16T02:29:20.547 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-08-16T02:29:20.547 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-16T02:29:20.609 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-16T02:29:20.610 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-16T02:29:20.677 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-16T02:29:20.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:29:20.821 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-16T02:29:20.830 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-16T02:29:20.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-16T02:29:20.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-16T02:29:20.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:29:20.921 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-08-16T02:29:20.942 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:29:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:29:20.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-16T02:29:21.007 INFO:teuthology.task.internal:Transferring archived files... 2024-08-16T02:29:21.008 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857404/remote/smithi067 2024-08-16T02:29:21.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-16T02:29:21.071 INFO:teuthology.task.internal:Removing archive directory... 2024-08-16T02:29:21.072 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-16T02:29:21.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-16T02:29:21.132 INFO:teuthology.task.internal:Not uploading archives. 2024-08-16T02:29:21.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-16T02:29:21.212 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-16T02:29:21.212 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-16T02:29:21.228 INFO:teuthology.orchestra.run.smithi067.stdout: 265318 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:29 /home/ubuntu/cephtest 2024-08-16T02:29:21.228 INFO:teuthology.orchestra.run.smithi067.stdout: 406091 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:19 /home/ubuntu/cephtest/mnt.0 2024-08-16T02:29:21.228 INFO:teuthology.orchestra.run.smithi067.stdout: 406092 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:20 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-16T02:29:21.228 INFO:teuthology.orchestra.run.smithi067.stdout: 420903 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:20 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-16T02:29:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:29:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-16T02:29:21.230 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-16T02:29:21.236 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-16T02:29:21.247 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-16T02:29:21.256 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi067 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-16T02:29:21.260 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format} duration: 1090.7657186985016 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi067 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2d0bcf10a075461cb543b68553df0467 status: fail success: false 2024-08-16T02:29:21.261 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-16T02:29:21.345 INFO:teuthology.run:FAIL