2024-09-22T02:49:52.006 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T02:49:52.022 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T02:49:52.119 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914717 branch: squid description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914717' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(POOL_FULL\) sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid env: CRIMSON_COMPAT: '1' sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1158 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9cxK7PZl+4RPWeRux28AjffN86ofORZrPRTRkGUdvWxFUdYeuEyFN8T+3Jdt91uSuc40pSm6hfYLK/sAf0u04= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T02:49:52.120 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-22T02:49:52.120 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-22T02:49:52.120 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T02:49:52.122 INFO:teuthology.task.internal:Checking packages... 2024-09-22T02:49:52.143 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-22T02:49:52.144 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T02:49:52.144 INFO:teuthology.packaging:ref: None 2024-09-22T02:49:52.144 INFO:teuthology.packaging:tag: None 2024-09-22T02:49:52.144 INFO:teuthology.packaging:branch: squid 2024-09-22T02:49:52.144 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T02:49:52.144 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-22T02:49:52.332 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-22T02:49:52.334 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T02:49:52.340 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T02:49:52.341 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T02:49:52.385 INFO:teuthology.task.internal:Saving configuration 2024-09-22T02:49:52.397 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T02:49:52.485 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T02:49:52.511 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 02:44:04.842525', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9cxK7PZl+4RPWeRux28AjffN86ofORZrPRTRkGUdvWxFUdYeuEyFN8T+3Jdt91uSuc40pSm6hfYLK/sAf0u04='} 2024-09-22T02:49:52.512 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T02:49:52.734 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-22T02:49:52.734 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T02:49:52.786 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faada0b0160>, signals=[15]) 2024-09-22T02:49:52.786 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T02:49:52.792 INFO:teuthology.task.internal:Opening connections... 2024-09-22T02:49:52.792 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T02:49:52.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:49:52.870 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T02:49:52.876 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-09-22T02:49:52.953 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-09-22T02:49:52.953 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-09-22T02:49:53.011 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2024-09-22T02:49:53.011 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2024-09-22T02:49:53.011 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2024-09-22T02:49:53.011 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2024-09-22T02:49:53.011 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T02:49:53.012 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T02:49:53.013 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-09-22T02:49:53.035 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T02:49:53.042 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T02:49:53.059 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T02:49:53.059 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-09-22T02:49:53.079 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T02:49:53.084 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T02:49:53.085 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-09-22T02:49:53.147 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T02:49:53.179 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T02:49:53.193 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T02:49:53.193 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T02:49:53.194 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T02:49:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T02:49:53.211 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-22T02:49:53.268 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-503.el9.x86_64 2024-09-22T02:49:53.269 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.14.0-503.el9.x86_64 2024-09-22T02:49:53.269 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2024-09-22T02:49:55.669 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-22T02:49:59.286 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 20 MB 00:02 2024-09-22T02:50:03.628 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-22T02:50:05.337 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-22T02:50:06.358 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-22T02:50:12.039 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-22T02:50:13.970 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T02:50:13.970 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T02:50:14.066 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T02:50:14.077 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T02:50:14.078 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:50:14.079 INFO:teuthology.orchestra.run.smithi039.stdout:Install 4 Packages 2024-09-22T02:50:14.079 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:50:14.079 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 79 M 2024-09-22T02:50:14.080 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 126 M 2024-09-22T02:50:14.080 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T02:50:14.240 INFO:teuthology.orchestra.run.smithi039.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 313 kB/s | 36 kB 00:00 2024-09-22T02:50:20.915 INFO:teuthology.orchestra.run.smithi039.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-09-22T02:50:27.173 INFO:teuthology.orchestra.run.smithi039.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.2 MB/s | 29 MB 00:12 2024-09-22T02:50:29.989 INFO:teuthology.orchestra.run.smithi039.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 35 MB 00:15 2024-09-22T02:50:29.989 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:50:29.990 INFO:teuthology.orchestra.run.smithi039.stdout:Total 5.0 MB/s | 79 MB 00:15 2024-09-22T02:50:30.701 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T02:50:31.064 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T02:50:31.064 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T02:50:32.106 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T02:50:32.107 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T02:50:34.680 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T02:50:35.325 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:50:35.406 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:50:38.150 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:50:38.286 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:50:43.407 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:50:43.454 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:50:48.595 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:51:27.473 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:51:27.514 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:51:27.915 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:51:27.916 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T02:51:27.916 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T02:51:27.916 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T02:51:28.927 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T02:51:28.928 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:51:28.928 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T02:51:28.928 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T02:51:28.928 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T02:51:28.928 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T02:51:28.928 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T02:51:28.928 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:51:28.928 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T02:51:29.273 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:51:29.363 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:51:29.364 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:51:29.364 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T02:51:29.364 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T02:51:29.364 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T02:51:29.364 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T02:51:29.364 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T02:51:29.364 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T02:51:29.423 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T02:51:29.423 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T02:51:29.423 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2024-09-22T02:51:29.462 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 82.el9 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8274670 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2024-09-22T02:51:29.463 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:51:29.464 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T02:51:29.465 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T02:51:29.465 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 5.14.0-508.el9.x86_64 2024-09-22T02:51:29.465 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T02:51:30.111 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-09-22T02:51:31.412 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T02:51:31.460 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-09-22T02:51:31.463 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-22T02:51:31.478 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.2Mm0ydDub4 2024-09-22T02:51:31.478 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2Mm0ydDub4 2024-09-22T02:51:31.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.2Mm0ydDub4 2024-09-22T02:51:31.677 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.2Mm0ydDub4 is 6KB 2024-09-22T02:51:31.725 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.2Mm0ydDub4 2024-09-22T02:51:31.740 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T02:51:31.802 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T02:51:31.802 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T02:51:31.802 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T02:51:31.802 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T02:51:31.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T02:51:31.958 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-09-22T02:52:01.990 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T02:52:01.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-22T02:52:01.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:52:20.335 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-22T02:52:29.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-22T02:52:29.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:52:32.403 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-22T02:52:44.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-22T02:52:44.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:52:47.475 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-22T02:53:02.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-22T02:53:02.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:54:02.527 DEBUG:teuthology.orchestra.remote:timed out 2024-09-22T02:54:20.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-22T02:54:20.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:54:20.793 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-22T02:54:21.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-22T02:54:21.151 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T02:54:21.152 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-22T02:54:21.168 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-508.el9.x86_64 2024-09-22T02:54:21.169 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T02:54:21.169 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T02:54:21.169 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T02:54:22.170 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T02:54:22.171 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T02:54:22.261 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-09-22T02:54:22.292 DEBUG:teuthology.parallel:result is None 2024-09-22T02:54:22.292 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T02:54:22.299 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T02:54:22.299 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T02:54:22.319 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T02:54:22.344 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T02:54:22.371 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T02:54:22.372 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T02:54:22.401 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T02:54:22.407 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T02:54:22.407 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T02:54:22.471 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T02:54:22.482 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T02:54:22.484 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T02:54:22.497 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T02:54:22.498 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T02:54:22.553 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T02:54:22.562 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T02:54:22.562 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T02:54:22.611 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T02:54:22.696 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T02:54:22.752 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T02:54:22.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T02:54:22.876 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T02:54:22.876 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T02:54:22.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-09-22T02:54:23.023 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T02:54:23.322 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T02:54:23.330 INFO:teuthology.task.internal:Starting timer... 2024-09-22T02:54:23.330 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T02:54:23.375 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T02:54:23.407 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-22T02:54:23.408 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2024-09-22T02:54:23.466 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2024-09-22T02:54:23.468 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T02:54:23.468 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2024-09-22T02:54:23.499 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2024-09-22T02:54:23.499 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive'} 2024-09-22T02:54:23.500 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T02:54:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T02:54:23.565 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T02:54:23.565 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2024-09-22T02:54:23.634 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T02:54:23.644 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T02:54:23.655 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T02:54:23.655 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T02:58:11.914 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com')] 2024-09-22T02:58:11.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-22T02:58:11.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T02:58:11.991 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-22T02:58:12.066 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-22T02:58:12.066 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T02:58:12.078 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T02:58:12.078 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T02:58:12.078 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T02:58:12.135 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T02:58:12.149 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T02:58:12.175 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2024-09-22T02:58:12.186 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2024-09-22T02:58:12.202 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T02:58:12.216 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T02:58:12.274 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-09-22T02:58:12.279 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T02:58:12.279 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-09-22T02:58:12.279 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:58:12.279 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:58:12.279 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:58:12.279 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T02:58:12.279 INFO:teuthology.run_tasks:Running task install... 2024-09-22T02:58:12.288 DEBUG:teuthology.task.install:project ceph 2024-09-22T02:58:12.288 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-22T02:58:12.288 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'} 2024-09-22T02:58:12.288 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T02:58:12.293 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T02:58:12.293 INFO:teuthology.task.install:extra packages: [] 2024-09-22T02:58:12.294 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-22T02:58:12.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T02:58:12.437 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/ 2024-09-22T02:58:12.438 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-22T02:58:12.557 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T02:58:12.557 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T02:58:12.557 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T02:58:12.586 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T02:58:12.587 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T02:58:12.654 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T02:58:12.731 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T02:58:12.797 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2024-09-22T02:58:12.799 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2024-09-22T02:58:13.043 INFO:teuthology.orchestra.run.smithi039.stdout:45 files removed 2024-09-22T02:58:13.070 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw 2024-09-22T02:58:13.643 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 317 kB/s | 87 kB 00:00 2024-09-22T02:58:13.919 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 51 kB/s | 12 kB 00:00 2024-09-22T02:58:14.178 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 9.0 kB/s | 2.0 kB 00:00 2024-09-22T02:58:16.345 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.3 MB 00:02 2024-09-22T02:58:18.154 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-22T02:58:23.880 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-22T02:58:25.398 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 98 kB/s | 19 kB 00:00 2024-09-22T02:58:26.360 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-22T02:58:32.099 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-22T02:58:34.438 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 9.9 M 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-22T02:58:34.455 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 19 M 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 760 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 501 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.1 M 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T02:58:34.456 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 161 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 301 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2024-09-22T02:58:34.457 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T02:58:34.458 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:58:34.458 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T02:58:34.458 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:58:34.458 INFO:teuthology.orchestra.run.smithi039.stdout:Install 30 Packages 2024-09-22T02:58:34.458 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2024-09-22T02:58:34.458 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:58:34.458 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 58 M 2024-09-22T02:58:34.458 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T02:58:35.943 INFO:teuthology.orchestra.run.smithi039.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 4.2 MB/s | 5.4 MB 00:01 2024-09-22T02:58:36.026 INFO:teuthology.orchestra.run.smithi039.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 303 kB/s | 25 kB 00:00 2024-09-22T02:58:36.193 INFO:teuthology.orchestra.run.smithi039.stdout:(3/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 4.5 MB/s | 760 kB 00:00 2024-09-22T02:58:36.293 INFO:teuthology.orchestra.run.smithi039.stdout:(4/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 4.9 MB/s | 501 kB 00:00 2024-09-22T02:58:36.635 INFO:teuthology.orchestra.run.smithi039.stdout:(5/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_64 15 MB/s | 5.1 MB 00:00 2024-09-22T02:58:36.869 INFO:teuthology.orchestra.run.smithi039.stdout:(6/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 4.5 MB/s | 9.9 MB 00:02 2024-09-22T02:58:36.894 INFO:teuthology.orchestra.run.smithi039.stdout:(7/32): python3-ceph-argparse-19.1.1-373.g4e45e 175 kB/s | 45 kB 00:00 2024-09-22T02:58:36.919 INFO:teuthology.orchestra.run.smithi039.stdout:(8/32): python3-ceph-common-19.1.1-373.g4e45e6b 2.7 MB/s | 136 kB 00:00 2024-09-22T02:58:36.945 INFO:teuthology.orchestra.run.smithi039.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 3.1 MB/s | 161 kB 00:00 2024-09-22T02:58:36.995 INFO:teuthology.orchestra.run.smithi039.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 4.2 MB/s | 321 kB 00:00 2024-09-22T02:58:37.020 INFO:teuthology.orchestra.run.smithi039.stdout:(11/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 3.9 MB/s | 301 kB 00:00 2024-09-22T02:58:37.045 INFO:teuthology.orchestra.run.smithi039.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-22T02:58:37.296 INFO:teuthology.orchestra.run.smithi039.stdout:(13/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x 7.4 MB/s | 19 MB 00:02 2024-09-22T02:58:37.321 INFO:teuthology.orchestra.run.smithi039.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 355 kB/s | 107 kB 00:00 2024-09-22T02:58:37.404 INFO:teuthology.orchestra.run.smithi039.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-22T02:58:37.430 INFO:teuthology.orchestra.run.smithi039.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 338 kB/s | 45 kB 00:00 2024-09-22T02:58:37.497 INFO:teuthology.orchestra.run.smithi039.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-22T02:58:37.522 INFO:teuthology.orchestra.run.smithi039.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 459 kB/s | 42 kB 00:00 2024-09-22T02:58:37.555 INFO:teuthology.orchestra.run.smithi039.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-22T02:58:37.581 INFO:teuthology.orchestra.run.smithi039.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 612 kB/s | 36 kB 00:00 2024-09-22T02:58:37.606 INFO:teuthology.orchestra.run.smithi039.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-22T02:58:37.632 INFO:teuthology.orchestra.run.smithi039.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-09-22T02:58:37.698 INFO:teuthology.orchestra.run.smithi039.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 511 kB/s | 26 kB 00:00 2024-09-22T02:58:37.749 INFO:teuthology.orchestra.run.smithi039.stdout:(24/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.8 MB/s | 308 kB 00:00 2024-09-22T02:58:37.774 INFO:teuthology.orchestra.run.smithi039.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 643 kB/s | 49 kB 00:00 2024-09-22T02:58:37.799 INFO:teuthology.orchestra.run.smithi039.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-22T02:58:37.841 INFO:teuthology.orchestra.run.smithi039.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-22T02:58:37.891 INFO:teuthology.orchestra.run.smithi039.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-22T02:58:38.092 INFO:teuthology.orchestra.run.smithi039.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-09-22T02:58:38.209 INFO:teuthology.orchestra.run.smithi039.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-09-22T02:58:38.267 INFO:teuthology.orchestra.run.smithi039.stdout:(31/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 8.5 MB/s | 3.2 MB 00:00 2024-09-22T02:58:38.384 INFO:teuthology.orchestra.run.smithi039.stdout:(32/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 11 MB/s | 3.1 MB 00:00 2024-09-22T02:58:38.390 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:58:38.390 INFO:teuthology.orchestra.run.smithi039.stdout:Total 15 MB/s | 58 MB 00:03 2024-09-22T02:58:38.499 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T02:58:38.540 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T02:58:38.541 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T02:58:39.245 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T02:58:39.246 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T02:58:40.845 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T02:58:41.247 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-22T02:58:41.370 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:58:41.463 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:58:41.556 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T02:58:41.596 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:58:41.674 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:58:41.709 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-09-22T02:58:41.734 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-09-22T02:58:41.757 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T02:58:41.785 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T02:58:41.969 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T02:58:42.004 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T02:58:42.033 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-09-22T02:58:42.078 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 10/34 2024-09-22T02:58:42.106 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T02:58:42.183 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-09-22T02:58:42.211 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:58:42.305 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:58:42.335 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-09-22T02:58:42.447 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-22T02:58:42.478 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-22T02:58:42.519 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-22T02:58:42.560 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-22T02:58:42.606 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-22T02:58:42.639 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-22T02:58:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-22T02:58:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T02:58:43.118 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T02:58:43.148 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-22T02:58:43.193 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-22T02:58:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-22T02:58:43.737 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T02:58:43.800 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T02:58:43.889 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-22T02:58:43.922 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-22T02:58:46.346 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:58:46.377 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:58:47.004 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T02:58:47.097 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T02:58:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T02:58:47.190 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T02:58:47.190 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T02:58:47.190 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:58:47.232 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:58:57.794 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:58:57.794 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2024-09-22T02:58:57.794 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2024-09-22T02:58:57.794 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2024-09-22T02:58:57.795 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2024-09-22T02:58:57.795 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2024-09-22T02:58:57.795 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2024-09-22T02:58:57.795 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2024-09-22T02:58:57.795 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:58:58.799 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T02:58:58.826 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T02:58:58.826 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T02:58:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T02:58:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T02:58:58.827 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T02:58:58.827 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:58:58.829 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T02:58:58.919 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T02:58:58.919 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-22T02:59:00.163 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-22T02:59:00.164 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-22T02:59:00.165 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-22T02:59:00.165 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-22T02:59:00.165 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-22T02:59:00.166 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-22T02:59:00.166 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T02:59:00.166 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-22T02:59:00.166 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-22T02:59:01.285 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T02:59:01.285 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:59:01.285 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-09-22T02:59:01.285 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.285 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.286 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T02:59:01.286 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T02:59:01.286 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.286 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.286 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.286 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.287 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T02:59:01.287 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T02:59:01.287 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T02:59:01.287 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.287 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T02:59:01.288 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T02:59:01.288 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T02:59:01.288 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.288 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T02:59:01.288 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.288 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T02:59:01.289 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T02:59:01.289 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T02:59:01.289 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.289 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.289 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.289 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T02:59:01.290 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.290 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.290 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:01.290 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T02:59:01.290 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T02:59:01.290 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T02:59:01.291 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T02:59:01.291 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T02:59:01.291 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:59:01.291 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T02:59:01.616 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-test 2024-09-22T02:59:02.218 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:30 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T02:59:02.356 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T02:59:02.357 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:59:02.357 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T02:59:02.357 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 124 M 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout: libasan x86_64 11.5.0-2.el9 appstream 414 k 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 158 k 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:59:02.358 INFO:teuthology.orchestra.run.smithi039.stdout:Install 7 Packages 2024-09-22T02:59:02.359 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:59:02.359 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 125 M 2024-09-22T02:59:02.359 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.1 G 2024-09-22T02:59:02.359 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T02:59:02.808 INFO:teuthology.orchestra.run.smithi039.stdout:(1/7): libasan-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 414 kB 00:00 2024-09-22T02:59:02.875 INFO:teuthology.orchestra.run.smithi039.stdout:(2/7): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 768 kB/s | 158 kB 00:00 2024-09-22T02:59:02.900 INFO:teuthology.orchestra.run.smithi039.stdout:(3/7): libubsan-11.5.0-2.el9.x86_64.rpm 1.9 MB/s | 183 kB 00:00 2024-09-22T02:59:02.984 INFO:teuthology.orchestra.run.smithi039.stdout:(4/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 764 kB/s | 64 kB 00:00 2024-09-22T02:59:03.042 INFO:teuthology.orchestra.run.smithi039.stdout:(5/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-09-22T02:59:03.067 INFO:teuthology.orchestra.run.smithi039.stdout:(6/7): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-22T02:59:06.226 INFO:teuthology.orchestra.run.smithi039.stdout:(7/7): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 35 MB/s | 124 MB 00:03 2024-09-22T02:59:06.228 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:59:06.229 INFO:teuthology.orchestra.run.smithi039.stdout:Total 32 MB/s | 125 MB 00:03 2024-09-22T02:59:06.284 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T02:59:06.307 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T02:59:06.308 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T02:59:07.171 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T02:59:07.171 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T02:59:08.277 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T02:59:08.369 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-09-22T02:59:08.418 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-09-22T02:59:08.456 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-09-22T02:59:08.494 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T02:59:08.544 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libasan-11.5.0-2.el9.x86_64 5/7 2024-09-22T02:59:08.571 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T02:59:26.725 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T02:59:26.803 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T02:59:35.144 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T02:59:35.144 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/7 2024-09-22T02:59:35.144 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/7 2024-09-22T02:59:35.144 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libasan-11.5.0-2.el9.x86_64 3/7 2024-09-22T02:59:35.145 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T02:59:35.145 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-09-22T02:59:35.145 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-09-22T02:59:35.551 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: libasan-11.5.0-2.el9.x86_64 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:59:35.552 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T02:59:35.874 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph 2024-09-22T02:59:36.478 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:04 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T02:59:36.619 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.3 M 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.9 M 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 120 M 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T02:59:36.622 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T02:59:36.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T02:59:36.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T02:59:36.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T02:59:36.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:59:36.625 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T02:59:36.625 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T02:59:36.625 INFO:teuthology.orchestra.run.smithi039.stdout:Install 38 Packages 2024-09-22T02:59:36.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T02:59:36.625 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 134 M 2024-09-22T02:59:36.625 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.4 G 2024-09-22T02:59:36.626 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T02:59:37.528 INFO:teuthology.orchestra.run.smithi039.stdout:(1/38): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 52 kB/s | 6.6 kB 00:00 2024-09-22T02:59:37.715 INFO:teuthology.orchestra.run.smithi039.stdout:(2/38): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 4.8 MB/s | 1.5 MB 00:00 2024-09-22T02:59:37.765 INFO:teuthology.orchestra.run.smithi039.stdout:(3/38): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 6.3 MB/s | 2.3 MB 00:00 2024-09-22T02:59:37.865 INFO:teuthology.orchestra.run.smithi039.stdout:(4/38): ceph-mgr-modules-core-19.1.1-373.g4e45e 2.4 MB/s | 249 kB 00:00 2024-09-22T02:59:37.941 INFO:teuthology.orchestra.run.smithi039.stdout:(5/38): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-09-22T02:59:38.108 INFO:teuthology.orchestra.run.smithi039.stdout:(6/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-22T02:59:38.167 INFO:teuthology.orchestra.run.smithi039.stdout:(7/38): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-22T02:59:38.250 INFO:teuthology.orchestra.run.smithi039.stdout:(8/38): python3-pycparser-2.20-6.el9.noarch.rpm 1.6 MB/s | 135 kB 00:00 2024-09-22T02:59:38.325 INFO:teuthology.orchestra.run.smithi039.stdout:(9/38): python3-requests-2.25.1-8.el9.noarch.rp 1.6 MB/s | 125 kB 00:00 2024-09-22T02:59:38.417 INFO:teuthology.orchestra.run.smithi039.stdout:(10/38): python3-cryptography-36.0.1-4.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-09-22T02:59:38.443 INFO:teuthology.orchestra.run.smithi039.stdout:(11/38): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-09-22T02:59:38.635 INFO:teuthology.orchestra.run.smithi039.stdout:(12/38): lua-5.4.4-4.el9.x86_64.rpm 869 kB/s | 188 kB 00:00 2024-09-22T02:59:38.718 INFO:teuthology.orchestra.run.smithi039.stdout:(13/38): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-22T02:59:38.760 INFO:teuthology.orchestra.run.smithi039.stdout:(14/38): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-09-22T02:59:38.802 INFO:teuthology.orchestra.run.smithi039.stdout:(15/38): python3-markupsafe-1.1.1-12.el9.x86_64 417 kB/s | 35 kB 00:00 2024-09-22T02:59:38.828 INFO:teuthology.orchestra.run.smithi039.stdout:(16/38): python3-packaging-20.9-5.el9.noarch.rp 1.1 MB/s | 77 kB 00:00 2024-09-22T02:59:38.853 INFO:teuthology.orchestra.run.smithi039.stdout:(17/38): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-22T02:59:38.878 INFO:teuthology.orchestra.run.smithi039.stdout:(18/38): python3-toml-0.10.2-6.el9.noarch.rpm 827 kB/s | 42 kB 00:00 2024-09-22T02:59:38.920 INFO:teuthology.orchestra.run.smithi039.stdout:(19/38): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-22T02:59:38.946 INFO:teuthology.orchestra.run.smithi039.stdout:(20/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-22T02:59:38.971 INFO:teuthology.orchestra.run.smithi039.stdout:(21/38): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-09-22T02:59:38.996 INFO:teuthology.orchestra.run.smithi039.stdout:(22/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-22T02:59:39.030 INFO:teuthology.orchestra.run.smithi039.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 406 kB/s | 11 kB 00:00 2024-09-22T02:59:39.055 INFO:teuthology.orchestra.run.smithi039.stdout:(24/38): python3-jaraco-classes-3.2.1-5.el9.noa 764 kB/s | 18 kB 00:00 2024-09-22T02:59:39.080 INFO:teuthology.orchestra.run.smithi039.stdout:(25/38): python3-jaraco-collections-3.0.0-8.el9 930 kB/s | 23 kB 00:00 2024-09-22T02:59:39.106 INFO:teuthology.orchestra.run.smithi039.stdout:(26/38): python3-jaraco-functools-3.5.0-2.el9.n 781 kB/s | 19 kB 00:00 2024-09-22T02:59:39.131 INFO:teuthology.orchestra.run.smithi039.stdout:(27/38): python3-jaraco-text-3.2.0-6.el9.noarch 789 kB/s | 20 kB 00:00 2024-09-22T02:59:39.156 INFO:teuthology.orchestra.run.smithi039.stdout:(28/38): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-22T02:59:39.181 INFO:teuthology.orchestra.run.smithi039.stdout:(29/38): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-22T02:59:39.207 INFO:teuthology.orchestra.run.smithi039.stdout:(30/38): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-22T02:59:39.232 INFO:teuthology.orchestra.run.smithi039.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 662 kB/s | 16 kB 00:00 2024-09-22T02:59:39.257 INFO:teuthology.orchestra.run.smithi039.stdout:(32/38): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-22T02:59:39.283 INFO:teuthology.orchestra.run.smithi039.stdout:(33/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-22T02:59:39.308 INFO:teuthology.orchestra.run.smithi039.stdout:(34/38): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-22T02:59:39.333 INFO:teuthology.orchestra.run.smithi039.stdout:(35/38): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-22T02:59:39.358 INFO:teuthology.orchestra.run.smithi039.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-22T02:59:39.384 INFO:teuthology.orchestra.run.smithi039.stdout:(37/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-22T02:59:41.434 INFO:teuthology.orchestra.run.smithi039.stdout:(38/38): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 32 MB/s | 120 MB 00:03 2024-09-22T02:59:41.438 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T02:59:41.438 INFO:teuthology.orchestra.run.smithi039.stdout:Total 28 MB/s | 134 MB 00:04 2024-09-22T02:59:41.614 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T02:59:41.657 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T02:59:41.658 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T02:59:42.789 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T02:59:42.790 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T02:59:44.829 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T02:59:44.903 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-09-22T02:59:44.942 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-09-22T02:59:44.973 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-09-22T02:59:45.048 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-09-22T02:59:45.089 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-09-22T02:59:45.121 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-09-22T02:59:45.217 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-09-22T02:59:45.302 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-09-22T02:59:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/38 2024-09-22T02:59:45.384 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-09-22T02:59:45.422 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-09-22T02:59:45.465 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-09-22T02:59:45.499 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-09-22T02:59:45.540 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-09-22T02:59:45.583 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-09-22T02:59:45.617 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-09-22T02:59:45.667 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-09-22T02:59:45.723 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-09-22T02:59:45.807 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-09-22T02:59:45.958 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-09-22T02:59:46.011 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-09-22T03:00:06.078 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-09-22T03:00:06.108 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:00:06.152 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:00:06.152 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:00:06.152 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:00:06.152 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:00:06.152 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:00:06.153 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:06.259 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-09-22T03:00:06.305 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-09-22T03:00:06.363 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-09-22T03:00:06.427 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-09-22T03:00:06.623 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-09-22T03:00:06.669 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-09-22T03:00:06.749 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-09-22T03:00:06.848 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-09-22T03:00:06.893 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-09-22T03:00:07.016 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-09-22T03:00:07.185 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 34/38 2024-09-22T03:00:07.212 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:00:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:00:07.240 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:00:07.240 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:00:07.240 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:00:07.240 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:00:07.240 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:07.729 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:00:07.755 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:00:07.755 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:00:07.756 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:00:07.756 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:00:07.756 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:00:07.756 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:07.996 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:00:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:00:08.022 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:00:08.022 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:00:08.022 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:00:08.022 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:00:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:00:18.907 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:00:18.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/38 2024-09-22T03:00:18.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/38 2024-09-22T03:00:18.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/38 2024-09-22T03:00:18.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/38 2024-09-22T03:00:18.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/38 2024-09-22T03:00:18.909 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/38 2024-09-22T03:00:18.909 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-09-22T03:00:18.909 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-09-22T03:00:18.909 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-09-22T03:00:18.909 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-09-22T03:00:18.909 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-09-22T03:00:18.910 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-09-22T03:00:18.910 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-09-22T03:00:18.910 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-09-22T03:00:18.910 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-09-22T03:00:18.910 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-09-22T03:00:18.910 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-09-22T03:00:18.910 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-09-22T03:00:18.911 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-09-22T03:00:18.911 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-09-22T03:00:18.911 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-09-22T03:00:18.911 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-09-22T03:00:18.911 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-09-22T03:00:18.911 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-09-22T03:00:18.912 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-09-22T03:00:18.912 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-09-22T03:00:18.912 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-09-22T03:00:18.912 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-09-22T03:00:18.912 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-09-22T03:00:18.912 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-09-22T03:00:18.913 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-09-22T03:00:18.913 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-09-22T03:00:18.913 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-09-22T03:00:18.913 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-09-22T03:00:18.913 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-09-22T03:00:18.913 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 36/38 2024-09-22T03:00:18.914 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-09-22T03:00:19.789 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-09-22T03:00:19.789 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:19.789 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:19.789 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:00:19.789 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:00:19.790 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:00:19.791 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:19.792 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:20.064 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-base 2024-09-22T03:00:20.672 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:48 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:20.764 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:00:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:20.809 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:00:20.810 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:20.885 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install cephadm 2024-09-22T03:00:21.491 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:49 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:21.627 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:21.628 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:21.628 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T03:00:21.628 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:21.628 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:21.628 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-22T03:00:21.628 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:21.628 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:21.628 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:21.629 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-22T03:00:21.629 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:21.629 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 765 k 2024-09-22T03:00:21.629 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 771 k 2024-09-22T03:00:21.629 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:21.898 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-22T03:00:21.898 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:21.899 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-22T03:00:21.899 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:21.905 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:21.905 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:21.911 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:21.911 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:00:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:00:22.427 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:00:22.538 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:00:22.914 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:00:23.166 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:00:23.166 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:23.166 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:23.167 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:23.167 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:23.167 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:23.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:00:23.981 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:51 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:24.123 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:24.124 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:24.124 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-09-22T03:00:24.124 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:24.124 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:24.124 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 147 k 2024-09-22T03:00:24.124 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:24.125 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:24.125 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:24.125 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-22T03:00:24.125 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:24.125 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 147 k 2024-09-22T03:00:24.125 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 481 k 2024-09-22T03:00:24.126 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:24.337 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 691 kB/s | 147 kB 00:00 2024-09-22T03:00:24.338 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:24.338 INFO:teuthology.orchestra.run.smithi039.stdout:Total 687 kB/s | 147 kB 00:00 2024-09-22T03:00:24.338 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:24.346 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:24.406 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:24.407 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:00:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:00:24.622 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:00:24.648 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:00:24.648 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:00:24.648 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:00:24.648 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:00:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:25.633 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:00:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:25.633 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:25.887 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr 2024-09-22T03:00:26.497 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:54 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:26.591 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:00:26.636 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:26.637 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:00:26.637 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:26.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:00:27.392 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:55 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:27.530 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:00:27.532 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:00:27.533 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:00:27.533 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:00:27.533 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:00:27.533 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:00:27.533 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:27.533 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:27.533 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:27.534 INFO:teuthology.orchestra.run.smithi039.stdout:Install 11 Packages 2024-09-22T03:00:27.534 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:27.535 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 7.7 M 2024-09-22T03:00:27.535 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 101 M 2024-09-22T03:00:27.535 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:27.824 INFO:teuthology.orchestra.run.smithi039.stdout:(1/11): ceph-prometheus-alerts-19.1.1-373.g4e45 102 kB/s | 16 kB 00:00 2024-09-22T03:00:27.849 INFO:teuthology.orchestra.run.smithi039.stdout:(2/11): ceph-grafana-dashboards-19.1.1-373.g4e4 171 kB/s | 32 kB 00:00 2024-09-22T03:00:28.050 INFO:teuthology.orchestra.run.smithi039.stdout:(3/11): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 9.2 MB/s | 3.6 MB 00:00 2024-09-22T03:00:28.075 INFO:teuthology.orchestra.run.smithi039.stdout:(4/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-09-22T03:00:28.117 INFO:teuthology.orchestra.run.smithi039.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 467 kB/s | 19 kB 00:00 2024-09-22T03:00:28.150 INFO:teuthology.orchestra.run.smithi039.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.3 MB/s | 548 kB 00:00 2024-09-22T03:00:28.184 INFO:teuthology.orchestra.run.smithi039.stdout:(7/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-09-22T03:00:28.218 INFO:teuthology.orchestra.run.smithi039.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-09-22T03:00:28.260 INFO:teuthology.orchestra.run.smithi039.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 5.4 MB/s | 188 kB 00:00 2024-09-22T03:00:28.285 INFO:teuthology.orchestra.run.smithi039.stdout:(10/11): python3-repoze-lru-0.7-16.el9.noarch.r 305 kB/s | 31 kB 00:00 2024-09-22T03:00:28.335 INFO:teuthology.orchestra.run.smithi039.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.3 MB/s | 2.0 MB 00:00 2024-09-22T03:00:28.340 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:28.340 INFO:teuthology.orchestra.run.smithi039.stdout:Total 9.5 MB/s | 7.7 MB 00:00 2024-09-22T03:00:28.370 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:28.389 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:28.595 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:28.596 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:00:29.099 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:00:29.193 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:00:29.222 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:00:29.360 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:00:29.490 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:00:29.542 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:00:29.628 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:00:29.678 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:00:29.705 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:00:29.738 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 9/11 2024-09-22T03:00:30.857 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 10/11 2024-09-22T03:00:30.905 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:00:32.258 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:00:32.259 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/11 2024-09-22T03:00:32.259 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/11 2024-09-22T03:00:32.259 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/11 2024-09-22T03:00:32.259 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:00:32.259 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:00:32.259 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:00:32.259 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:00:32.259 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:00:32.260 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:00:32.260 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:00:32.588 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:00:32.589 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:00:32.589 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:32.589 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:32.848 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:00:33.458 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:01 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:33.596 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================== 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================== 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:00:33.597 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================== 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout:Install 12 Packages 2024-09-22T03:00:33.598 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:33.599 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 38 M 2024-09-22T03:00:33.599 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 212 M 2024-09-22T03:00:33.599 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:33.792 INFO:teuthology.orchestra.run.smithi039.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-22T03:00:33.834 INFO:teuthology.orchestra.run.smithi039.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.4 MB/s | 801 kB 00:00 2024-09-22T03:00:33.884 INFO:teuthology.orchestra.run.smithi039.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 354 kB/s | 33 kB 00:00 2024-09-22T03:00:33.926 INFO:teuthology.orchestra.run.smithi039.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 435 kB/s | 18 kB 00:00 2024-09-22T03:00:33.968 INFO:teuthology.orchestra.run.smithi039.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 926 kB/s | 38 kB 00:00 2024-09-22T03:00:34.110 INFO:teuthology.orchestra.run.smithi039.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-09-22T03:00:34.211 INFO:teuthology.orchestra.run.smithi039.stdout:(7/12): ceph-mgr-diskprediction-local-19.1.1-37 13 MB/s | 7.4 MB 00:00 2024-09-22T03:00:34.252 INFO:teuthology.orchestra.run.smithi039.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-22T03:00:34.336 INFO:teuthology.orchestra.run.smithi039.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2024-09-22T03:00:34.362 INFO:teuthology.orchestra.run.smithi039.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.3 MB/s | 250 kB 00:00 2024-09-22T03:00:34.587 INFO:teuthology.orchestra.run.smithi039.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-09-22T03:00:34.820 INFO:teuthology.orchestra.run.smithi039.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-22T03:00:34.821 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:34.822 INFO:teuthology.orchestra.run.smithi039.stdout:Total 31 MB/s | 38 MB 00:01 2024-09-22T03:00:35.187 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:35.215 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:35.215 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:35.664 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:35.664 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:00:36.455 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:00:36.546 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:00:36.585 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:00:36.690 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:00:36.729 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:00:36.919 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:00:36.952 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:00:37.039 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:00:37.736 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:00:37.861 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:00:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:00:40.281 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:00:40.309 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:00:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:00:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:00:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:00:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:00:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:43.438 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:00:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:00:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:43.439 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:43.857 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:00:44.464 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:12 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:44.602 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:44.603 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:44.603 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T03:00:44.603 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:44.603 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:44.603 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-22T03:00:44.603 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T03:00:44.603 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:44.604 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:44.605 INFO:teuthology.orchestra.run.smithi039.stdout:Install 13 Packages 2024-09-22T03:00:44.605 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:44.605 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 2.2 M 2024-09-22T03:00:44.605 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 26 M 2024-09-22T03:00:44.605 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:44.802 INFO:teuthology.orchestra.run.smithi039.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 291 kB/s | 26 kB 00:00 2024-09-22T03:00:44.835 INFO:teuthology.orchestra.run.smithi039.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 157 kB/s | 19 kB 00:00 2024-09-22T03:00:44.944 INFO:teuthology.orchestra.run.smithi039.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-22T03:00:44.969 INFO:teuthology.orchestra.run.smithi039.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-22T03:00:45.019 INFO:teuthology.orchestra.run.smithi039.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 1.1 MB/s | 54 kB 00:00 2024-09-22T03:00:45.070 INFO:teuthology.orchestra.run.smithi039.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 643 kB/s | 32 kB 00:00 2024-09-22T03:00:45.099 INFO:teuthology.orchestra.run.smithi039.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-09-22T03:00:45.120 INFO:teuthology.orchestra.run.smithi039.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 281 kB/s | 14 kB 00:00 2024-09-22T03:00:45.154 INFO:teuthology.orchestra.run.smithi039.stdout:(9/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 112 kB/s | 49 kB 00:00 2024-09-22T03:00:45.204 INFO:teuthology.orchestra.run.smithi039.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-22T03:00:45.229 INFO:teuthology.orchestra.run.smithi039.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 806 kB/s | 59 kB 00:00 2024-09-22T03:00:45.255 INFO:teuthology.orchestra.run.smithi039.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-22T03:00:45.296 INFO:teuthology.orchestra.run.smithi039.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.0 MB/s | 1.0 MB 00:00 2024-09-22T03:00:45.302 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:45.302 INFO:teuthology.orchestra.run.smithi039.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-22T03:00:45.366 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:45.382 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:45.383 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:45.481 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:45.482 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:00:46.001 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:00:46.177 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:00:46.241 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:00:46.306 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:00:46.341 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:00:46.383 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:00:46.498 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:00:46.618 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:00:46.677 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:00:47.444 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:00:47.498 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:00:47.532 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:00:47.581 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:00:47.610 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:00:48.389 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:00:48.389 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:00:48.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:00:48.715 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:00:48.715 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:48.715 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:48.715 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:48.715 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:00:48.715 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:00:48.715 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:48.716 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:48.995 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:00:49.597 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:17 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:49.736 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:49.737 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T03:00:49.737 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:49.737 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-22T03:00:49.737 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T03:00:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:00:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:00:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:00:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:00:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:00:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:49.738 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2024-09-22T03:00:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:49.739 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 7.0 M 2024-09-22T03:00:49.739 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 33 M 2024-09-22T03:00:49.739 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:50.124 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-09-22T03:00:50.191 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-22T03:00:50.216 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-22T03:00:50.258 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 565 kB/s | 146 kB 00:00 2024-09-22T03:00:50.283 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-22T03:00:50.392 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-22T03:00:50.393 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:50.394 INFO:teuthology.orchestra.run.smithi039.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-22T03:00:50.466 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:50.478 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:50.478 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:50.586 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:50.586 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:00:50.914 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:00:51.098 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:00:51.162 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:00:51.714 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:00:51.808 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:00:51.880 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:00:51.906 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:00:52.624 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:00:52.624 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-22T03:00:52.624 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:00:52.624 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:00:52.624 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:00:52.625 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:00:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:52.905 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:53.173 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-fuse 2024-09-22T03:00:53.783 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:21 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:53.925 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:53.926 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:53.926 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T03:00:53.926 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:53.926 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:53.926 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 868 k 2024-09-22T03:00:53.926 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-22T03:00:53.926 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:00:53.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:53.927 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:53.927 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:53.927 INFO:teuthology.orchestra.run.smithi039.stdout:Install 2 Packages 2024-09-22T03:00:53.927 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:53.927 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 948 k 2024-09-22T03:00:53.927 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 3.1 M 2024-09-22T03:00:53.928 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 629 kB/s | 80 kB 00:00 2024-09-22T03:00:54.278 INFO:teuthology.orchestra.run.smithi039.stdout:(2/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 2.8 MB/s | 868 kB 00:00 2024-09-22T03:00:54.278 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:54.278 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.6 MB/s | 948 kB 00:00 2024-09-22T03:00:54.294 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:54.316 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:54.316 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:54.402 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:54.402 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:00:54.589 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:00:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:00:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:00:55.462 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:00:55.463 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-22T03:00:55.741 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:00:55.742 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:55.742 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:55.742 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:00:55.742 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:55.742 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:56.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-volume 2024-09-22T03:00:56.610 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:24 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:56.749 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:56.750 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-22T03:00:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:56.751 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 285 k 2024-09-22T03:00:56.751 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.3 M 2024-09-22T03:00:56.751 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:57.224 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 603 kB/s | 285 kB 00:00 2024-09-22T03:00:57.224 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:57.225 INFO:teuthology.orchestra.run.smithi039.stdout:Total 601 kB/s | 285 kB 00:00 2024-09-22T03:00:57.225 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:57.234 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:57.234 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:57.263 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:57.264 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:00:57.494 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:00:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:00:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:00:57.584 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:00:57.584 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:00:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:58.563 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:00:58.563 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:58.563 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:00:58.563 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:00:58.563 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:58.563 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:00:58.851 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados-devel 2024-09-22T03:00:59.462 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:27 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:00:59.600 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:00:59.600 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:59.600 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-09-22T03:00:59.600 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:59.600 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:00:59.600 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-22T03:00:59.600 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:59.601 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:00:59.601 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:00:59.601 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-22T03:00:59.601 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:00:59.601 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 127 k 2024-09-22T03:00:59.601 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 457 k 2024-09-22T03:00:59.601 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:00:59.808 INFO:teuthology.orchestra.run.smithi039.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 614 kB/s | 127 kB 00:00 2024-09-22T03:00:59.808 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:00:59.809 INFO:teuthology.orchestra.run.smithi039.stdout:Total 610 kB/s | 127 kB 00:00 2024-09-22T03:00:59.809 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:00:59.815 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:00:59.815 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:00:59.882 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:00:59.883 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:01:00.045 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:01:00.165 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:00.614 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:00.903 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:00.903 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:00.904 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:01:00.904 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:01:00.904 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:00.904 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:01.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs2 2024-09-22T03:01:01.743 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:29 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:01.836 INFO:teuthology.orchestra.run.smithi039.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:01:01.882 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:01.883 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:01:01.884 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:01.959 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs-devel 2024-09-22T03:01:02.562 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:02.700 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:02.701 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:02.701 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-09-22T03:01:02.701 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:02.701 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:01:02.701 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-22T03:01:02.702 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:02.702 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:01:02.702 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:02.702 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-22T03:01:02.702 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:02.702 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 32 k 2024-09-22T03:01:02.702 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 142 k 2024-09-22T03:01:02.702 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:01:02.858 INFO:teuthology.orchestra.run.smithi039.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 204 kB/s | 32 kB 00:00 2024-09-22T03:01:02.858 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:01:02.858 INFO:teuthology.orchestra.run.smithi039.stdout:Total 202 kB/s | 32 kB 00:00 2024-09-22T03:01:02.859 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:01:02.862 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:01:02.862 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:01:02.879 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:01:02.879 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:01:02.972 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:01:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:03.470 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:03.723 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:03.723 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:03.723 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:01:03.723 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:01:03.723 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:03.723 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:03.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados2 2024-09-22T03:01:04.522 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:32 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:04.615 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:01:04.662 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:04.663 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:01:04.663 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:04.734 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librbd1 2024-09-22T03:01:05.343 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:33 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:05.437 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:01:05.483 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:05.484 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:01:05.484 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:05.549 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rados 2024-09-22T03:01:06.152 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:34 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:06.246 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:01:06.292 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:06.293 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:01:06.293 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:06.366 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rgw 2024-09-22T03:01:06.968 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:34 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:07.059 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:01:07.106 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:07.107 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:01:07.107 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:07.174 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-cephfs 2024-09-22T03:01:07.775 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:35 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:07.868 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:01:07.914 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:07.915 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:01:07.915 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:07.982 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rbd 2024-09-22T03:01:08.588 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:36 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:08.680 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:01:08.727 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:08.728 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-22T03:01:08.729 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:08.798 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-fuse 2024-09-22T03:01:09.405 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:37 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:09.546 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-22T03:01:09.547 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:09.548 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 85 k 2024-09-22T03:01:09.548 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 256 k 2024-09-22T03:01:09.548 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:01:09.736 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 456 kB/s | 85 kB 00:00 2024-09-22T03:01:09.736 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:01:09.736 INFO:teuthology.orchestra.run.smithi039.stdout:Total 453 kB/s | 85 kB 00:00 2024-09-22T03:01:09.737 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:01:09.744 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:01:09.837 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:01:09.838 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:01:09.838 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:01:09.963 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:01:10.069 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:10.493 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:10.762 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:10.762 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:10.762 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:01:10.762 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:01:10.762 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:10.762 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:11.028 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-mirror 2024-09-22T03:01:11.645 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:39 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:11.787 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:11.787 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:11.787 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-09-22T03:01:11.787 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:11.787 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:01:11.788 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:01:11.788 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:11.788 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:01:11.788 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:11.788 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-22T03:01:11.788 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:11.788 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 3.1 M 2024-09-22T03:01:11.788 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 14 M 2024-09-22T03:01:11.789 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:01:12.131 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 8.9 MB/s | 3.1 MB 00:00 2024-09-22T03:01:12.132 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:01:12.132 INFO:teuthology.orchestra.run.smithi039.stdout:Total 8.9 MB/s | 3.1 MB 00:00 2024-09-22T03:01:12.132 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:01:12.142 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:01:12.142 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:01:12.228 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:01:12.228 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:01:12.711 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:01:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:12.810 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:01:12.810 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:01:12.810 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:01:12.810 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:01:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:13.875 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:13.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:13.875 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:01:13.875 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:01:13.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:13.875 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:14.118 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-nbd 2024-09-22T03:01:14.731 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:42 ago on Sun 22 Sep 2024 02:58:32 AM UTC. 2024-09-22T03:01:14.873 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 169 k 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-22T03:01:14.874 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:14.875 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 169 k 2024-09-22T03:01:14.875 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 540 k 2024-09-22T03:01:14.875 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-22T03:01:15.077 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 840 kB/s | 169 kB 00:00 2024-09-22T03:01:15.084 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:01:15.084 INFO:teuthology.orchestra.run.smithi039.stdout:Total 833 kB/s | 169 kB 00:00 2024-09-22T03:01:15.084 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-22T03:01:15.086 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-22T03:01:15.086 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-22T03:01:15.156 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-22T03:01:15.156 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-22T03:01:15.316 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-22T03:01:15.420 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:01:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:16.130 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-22T03:01:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:01:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:16.131 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-22T03:01:16.407 DEBUG:teuthology.parallel:result is None 2024-09-22T03:01:16.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:01:16.564 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:01:16.587 INFO:teuthology.orchestra.run.smithi039.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-22T03:01:16.588 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-22T03:01:16.588 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-22T03:01:16.589 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T03:01:16.590 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:16.590 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:01:16.654 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T03:01:16.654 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:16.655 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:01:16.720 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:01:16.786 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T03:01:16.787 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:16.787 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:01:16.852 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:01:16.917 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T03:01:16.918 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:16.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:01:16.984 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:01:17.049 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T03:01:17.137 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T03:01:17.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2024-09-22T03:01:17.199 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T03:01:17.199 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:01:17.224 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T03:01:17.224 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:01:17.296 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T03:01:17.297 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T03:01:17.297 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914717', 'branch': 'squid', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914717', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'squid', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1158, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9cxK7PZl+4RPWeRux28AjffN86ofORZrPRTRkGUdvWxFUdYeuEyFN8T+3Jdt91uSuc40pSm6hfYLK/sAf0u04='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T03:01:17.297 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:01:17.346 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:01:17.412 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:17.412 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:01:17.469 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:01:17.469 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-09-22T03:01:17.525 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:01:17.525 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:01:17.525 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-22T03:01:17.525 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:01:17.525 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:01:17.525 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-22 03:01:13.049188046 +0000 2024-09-22T03:01:17.526 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-22 02:57:52.377777977 +0000 2024-09-22T03:01:17.526 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-22 02:57:52.377777977 +0000 2024-09-22T03:01:17.526 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-22 02:57:52.377777977 +0000 2024-09-22T03:01:17.526 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:01:17.589 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-22T03:01:17.589 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-22T03:01:17.589 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000293468 s, 1.7 MB/s 2024-09-22T03:01:17.590 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:01:17.648 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-09-22T03:01:17.704 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:01:17.704 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:01:17.704 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-22T03:01:17.704 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:01:17.705 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:01:17.705 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-22 03:01:13.049188046 +0000 2024-09-22T03:01:17.705 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-22 02:57:52.724770823 +0000 2024-09-22T03:01:17.705 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-22 02:57:52.724770823 +0000 2024-09-22T03:01:17.705 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-22 02:57:52.724770823 +0000 2024-09-22T03:01:17.705 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:01:17.768 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-22T03:01:17.768 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-22T03:01:17.768 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000263416 s, 1.9 MB/s 2024-09-22T03:01:17.769 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:01:17.825 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-22 03:01:13.050188025 +0000 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-22 02:57:53.077763545 +0000 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-22 02:57:53.077763545 +0000 2024-09-22T03:01:17.882 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-22 02:57:53.077763545 +0000 2024-09-22T03:01:17.883 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:01:17.946 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-22T03:01:17.946 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-22T03:01:17.946 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000262106 s, 2.0 MB/s 2024-09-22T03:01:17.947 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:01:18.004 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-22 03:01:13.050188025 +0000 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-22 02:57:53.418756515 +0000 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-22 02:57:53.418756515 +0000 2024-09-22T03:01:18.060 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-22 02:57:53.418756515 +0000 2024-09-22T03:01:18.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:01:18.126 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-22T03:01:18.126 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-22T03:01:18.126 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000298865 s, 1.7 MB/s 2024-09-22T03:01:18.127 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:01:18.186 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:01:18.186 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:01:18.186 INFO:tasks.ceph:Generating config... 2024-09-22T03:01:18.187 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-22T03:01:18.188 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-22T03:01:18.188 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-22T03:01:18.188 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-22T03:01:18.188 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-22T03:01:18.188 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-22T03:01:18.188 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-22T03:01:18.188 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-22T03:01:18.188 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T03:01:18.188 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T03:01:18.188 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T03:01:18.188 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T03:01:18.188 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-22T03:01:18.189 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-22T03:01:18.189 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-22T03:01:18.189 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T03:01:18.189 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:01:18.189 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T03:01:18.189 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T03:01:18.189 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T03:01:18.189 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T03:01:18.189 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T03:01:18.189 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T03:01:18.189 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T03:01:18.189 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T03:01:18.189 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-22T03:01:18.189 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-22T03:01:18.190 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-22T03:01:18.190 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T03:01:18.190 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T03:01:18.297 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T03:01:18.300 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T03:01:18.349 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:01:18.374 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.39')] 2024-09-22T03:01:18.375 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.39', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-22T03:01:18.376 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:18.376 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:01:18.430 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.39 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:01:18.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:18.516+0000 7febfc674e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:18.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:18.516+0000 7febfc674e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:18.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:18.516+0000 7febfc674e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:18.518 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:01:18.518 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: generated fsid df5703b9-69b9-43c4-b0b0-b568274428f6 2024-09-22T03:01:18.518 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = quincy 2024-09-22T03:01:18.519 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 0 2024-09-22T03:01:18.519 INFO:teuthology.orchestra.run.smithi039.stdout:fsid df5703b9-69b9-43c4-b0b0-b568274428f6 2024-09-22T03:01:18.519 INFO:teuthology.orchestra.run.smithi039.stdout:last_changed 2024-09-22T03:01:18.517488+0000 2024-09-22T03:01:18.519 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-09-22T03:01:18.517488+0000 2024-09-22T03:01:18.519 INFO:teuthology.orchestra.run.smithi039.stdout:min_mon_release 17 (quincy) 2024-09-22T03:01:18.519 INFO:teuthology.orchestra.run.smithi039.stdout:election_strategy: 1 2024-09-22T03:01:18.519 INFO:teuthology.orchestra.run.smithi039.stdout:0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-22T03:01:18.519 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T03:01:18.520 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:01:18.535 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID df5703b9-69b9-43c4-b0b0-b568274428f6... 2024-09-22T03:01:18.536 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:01:18.619 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.620 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:01:18.621 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:01:18.622 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-09-22T03:01:18.623 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = 172.21.15.39 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: ms client mode = crc 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: ms cluster mode = crc 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon client mode = crc 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon cluster mode = crc 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon service mode = crc 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: ms service mode = crc 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = df5703b9-69b9-43c4-b0b0-b568274428f6 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-09-22T03:01:18.624 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:01:18.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: debug monc = 20 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: crimson osd obc lru size = 10 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-22T03:01:18.626 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = seastore 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: osd shutdown pgref assert = True 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-22T03:01:18.627 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:01:18.628 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: mon min osdmap epochs = 50 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune interval = 2 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune min = 15 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune txsize = 2 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: paxos service trim min = 10 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default crimson = True 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-09-22T03:01:18.629 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout: crimson seastar cpu cores = 0-1 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout: crimson seastar cpu cores = 2-3 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2024-09-22T03:01:18.630 INFO:teuthology.orchestra.run.smithi039.stdout: crimson seastar cpu cores = 4-5 2024-09-22T03:01:18.631 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T03:01:18.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T03:01:18.693 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T03:01:18.693 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:18.693 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T03:01:18.709 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:18.709 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T03:01:18.764 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T03:01:18.764 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:18.764 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:01:18.764 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:01:18.840 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:18.840 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:01:18.896 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T03:01:18.896 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T03:01:18.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:01:18.994 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:01:18.996 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T03:01:18.997 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T03:01:18.997 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T03:01:19.044 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T03:01:19.057 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T03:01:19.057 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:01:19.057 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T03:01:19.082 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:01:19.083 INFO:tasks.ceph:role: osd.0 2024-09-22T03:01:19.083 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T03:01:19.083 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:01:19.176 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:01:19.176 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:01:19.176 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:01:19.176 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:01:19.177 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:01:19.177 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:01:19.177 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:01:19.177 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:01:19.177 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:01:19.177 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:01:20.494 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-22T03:01:20.496 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-22T03:01:20.496 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T03:01:20.537 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T03:01:20.613 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T03:01:20.680 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:01:20.681 INFO:tasks.ceph:role: osd.1 2024-09-22T03:01:20.681 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T03:01:20.681 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:01:20.776 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:01:20.777 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:01:22.096 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-22T03:01:22.098 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-22T03:01:22.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T03:01:22.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T03:01:22.211 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T03:01:22.279 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:01:22.279 INFO:tasks.ceph:role: osd.2 2024-09-22T03:01:22.279 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T03:01:22.279 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:01:22.374 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:01:23.692 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-22T03:01:23.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-22T03:01:23.695 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-22T03:01:23.732 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T03:01:23.806 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:01:24.442 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:01:24.443 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:24,442 seastar - Reactor backend: linux-aio 2024-09-22T03:01:24.461 INFO:teuthology.orchestra.run.smithi039.stderr:==27629==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:01:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:24,505 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:01:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:24,505 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-22T03:01:24.507 INFO:teuthology.orchestra.run.smithi039.stderr:DEBUG 2024-09-22 03:01:24,506 [shard 0:main] seastar - reactor::drain 2024-09-22T03:01:25.037 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:25.037 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2024-09-22T03:01:25.037 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2024-09-22T03:01:25.037 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2024-09-22T03:01:25.037 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:25.037 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-22T03:01:25.095 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:01:25.096 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:25,096 seastar - Reactor backend: linux-aio 2024-09-22T03:01:25.116 INFO:teuthology.orchestra.run.smithi039.stderr:==27627==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:01:25.278 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x3985d 0x39f1e 0x20784631 0x301458ff 0x3014b444 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:25.278 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:25.311 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2ff3536a 0x2ff36035 0x3014b5be 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:01:25.312 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x39f05 0x255cfd50 0x30145078 0x30145a3b 0x3014b444 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:25.397 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b80 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:01:25.397 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:25.450 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b69 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:25.450 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:25.613 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xd5a9a 0xd554b 0x2c64a 0x282da 0xb49e1 0x3004b125 0x3014aa5f 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:25.613 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:25.671 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b80 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:01:25.671 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:25.728 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b15 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:25.729 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:26.395 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:26.395 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2024-09-22T03:01:26.395 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2024-09-22T03:01:26.396 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2024-09-22T03:01:26.396 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:26.396 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-22T03:01:26.396 INFO:teuthology.orchestra.run.smithi039.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid df5703b9-69b9-43c4-b0b0-b568274428f6 2024-09-22T03:01:26.441 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T03:01:26.467 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:01:27.074 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:01:27.075 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:27,074 seastar - Reactor backend: linux-aio 2024-09-22T03:01:27.097 INFO:teuthology.orchestra.run.smithi039.stderr:==27674==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:01:27.142 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:27,141 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:01:27.142 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:27,142 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-22T03:01:27.143 INFO:teuthology.orchestra.run.smithi039.stderr:DEBUG 2024-09-22 03:01:27,142 [shard 0:main] seastar - reactor::drain 2024-09-22T03:01:27.651 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:27.651 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2024-09-22T03:01:27.651 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2024-09-22T03:01:27.651 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2024-09-22T03:01:27.651 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:27.652 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-22T03:01:27.707 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:01:27.708 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:27,707 seastar - Reactor backend: linux-aio 2024-09-22T03:01:27.728 INFO:teuthology.orchestra.run.smithi039.stderr:==27672==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:01:27.890 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2bd7ec8d 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:27.891 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:27.923 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xd594a 0xd5ac0 0xd554b 0x287b7 0xb738c 0x24eabcb7 0x24eabd21 0x3014abaf 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:01:27.923 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:27.977 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c5f1 0x2857c 0xb6323 0x2fcc5109 0x2ffbd164 0x3014d7dd 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:27.978 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa718c637 0xffffffffa7196e56 0xffffffffa6e8c884 0xffffffffa7ad1db2 0xffffffffa7c00bb2 2024-09-22T03:01:28.009 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b2d 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:01:28.009 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:28.061 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095a78 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:28.061 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:28.222 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xd5a9a 0xd554b 0x287b7 0xb4674 0x2bd83fd2 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:28.222 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:28.279 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095a04 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:01:28.279 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095963 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:28.993 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:28.993 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2024-09-22T03:01:28.993 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2024-09-22T03:01:28.993 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2024-09-22T03:01:28.993 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:28.993 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-22T03:01:28.994 INFO:teuthology.orchestra.run.smithi039.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid df5703b9-69b9-43c4-b0b0-b568274428f6 2024-09-22T03:01:29.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T03:01:29.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:01:29.658 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:01:29.658 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:29,658 seastar - Reactor backend: linux-aio 2024-09-22T03:01:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:==27719==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:01:29.723 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:29,722 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:01:29.723 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:29,722 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-22T03:01:29.723 INFO:teuthology.orchestra.run.smithi039.stderr:DEBUG 2024-09-22 03:01:29,722 [shard 0:main] seastar - reactor::drain 2024-09-22T03:01:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2024-09-22T03:01:30.235 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2024-09-22T03:01:30.235 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2024-09-22T03:01:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:30.235 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-22T03:01:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:01:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2024-09-22 03:01:30,293 seastar - Reactor backend: linux-aio 2024-09-22T03:01:30.313 INFO:teuthology.orchestra.run.smithi039.stderr:==27717==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:01:30.476 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x30052412 0x30145156 0x30145a3b 0x3014b444 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:30.477 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:30.564 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x30151cc0 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:30.564 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095ad2 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:01:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:30.650 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b15 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:30.650 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:30.805 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x943d3 0x94d46 0x2ff768ac 0x2ff7730f 0x3014bdcc 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:01:30.805 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2024-09-22T03:01:31.595 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:31.595 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2024-09-22T03:01:31.596 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2024-09-22T03:01:31.596 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2024-09-22T03:01:31.596 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2024-09-22T03:01:31.596 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-22T03:01:31.596 INFO:teuthology.orchestra.run.smithi039.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid df5703b9-69b9-43c4-b0b0-b568274428f6 2024-09-22T03:01:31.642 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T03:01:31.668 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T03:01:31.668 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:31.668 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T03:01:31.733 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:31.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T03:01:31.799 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:31.800 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T03:01:31.865 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:31.865 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T03:01:31.931 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:31.931 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T03:01:31.987 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T03:01:31.987 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB+iO9mDr43OxAAQui1S4w1ml2QhDLF/4YC2g== 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCFiO9mDSZrChAAan6TwC1bvHDoTdGW05oEXg== 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCHiO9m7jDiLhAAXKlsooa13NFy9uoDwB82bg== 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCKiO9mmPsoFhAAj2eXEdTMF8IYvxhC/VcbXA== 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-09-22T03:01:32.051 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB/iO9mSrOaAhAA61fgLiD1EfgqdUckjg+nhQ== 2024-09-22T03:01:32.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T03:01:32.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:01:32.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:01:32.264 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:01:32.321 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:01:32.381 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T03:01:32.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T03:01:32.407 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T03:01:32.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:32.506+0000 7f8e5220fb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:32.506+0000 7f8e5220fb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.535 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T03:01:32.561 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:01:32.617 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T03:01:32.617 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T03:01:32.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T03:01:32.660 INFO:tasks.ceph.mon.a:Started 2024-09-22T03:01:32.660 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T03:01:32.661 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T03:01:32.661 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T03:01:32.663 INFO:tasks.ceph.mgr.x:Started 2024-09-22T03:01:32.663 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T03:01:32.663 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T03:01:32.741 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-22T03:01:32.739+0000 7f49a3010b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.741 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-22T03:01:32.739+0000 7f49a3010b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.741 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-22T03:01:32.739+0000 7f49a3010b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.742 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:32.740+0000 7f5188227f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:32.765+0000 7fcc4594e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:32.796+0000 7fcc4594e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.871 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2024-09-22T03:01:32.882 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T03:01:32.883 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T03:01:32.883 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T03:01:32.947 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:32.944+0000 7f5188227f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.947 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:32.945+0000 7f5188227f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:32.982+0000 7fa23704f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:32.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:32.989+0000 7fa23704f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:33.055 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:33.052+0000 7f5188227f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-22T03:01:33.073 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-22T03:01:33.071+0000 7f499aff1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T03:01:33.074 INFO:tasks.ceph.mon.a.smithi039.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T03:01:33.074 INFO:tasks.ceph.mon.a.smithi039.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T03:01:33.074 INFO:tasks.ceph.mon.a.smithi039.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T03:01:33.074 INFO:tasks.ceph.mon.a.smithi039.stderr:feature with important data. 2024-09-22T03:01:33.074 INFO:tasks.ceph.mon.a.smithi039.stderr: 2024-09-22T03:01:33.098 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:33.096+0000 7f5188227f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-22T03:01:33.240 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:33.238+0000 7f5188227f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-22T03:01:33.294 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:33.291+0000 7f5188227f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-22T03:01:33.453 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:33.451+0000 7f5188227f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:01:33.569 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:33.567+0000 7f5188227f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-22T03:01:33.884 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T03:01:33.884 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:33.884 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T03:01:33.909 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:33.910 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T03:01:33.916 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:33.914+0000 7f5188227f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-22T03:01:33.941 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:01:33.941 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T03:01:33.967 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:33.965+0000 7f5188227f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-22T03:01:33.992 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 1471c1f6-ee1d-4ea6-ae1f-1dd37999c2e7 0 2024-09-22T03:01:34.132 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:34.130+0000 7f5188227f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-22T03:01:34.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:34.131+0000 7fa0a646c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:34.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:34.140+0000 7fa0a646c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:34.225 INFO:teuthology.orchestra.run.smithi039.stdout:0 2024-09-22T03:01:34.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new f6b2e13d-3702-4ba6-b46b-cd4963228832 1 2024-09-22T03:01:34.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:34.335+0000 7f3fced57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:34.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:34.343+0000 7f3fced57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:34.424 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-09-22T03:01:34.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 2f051db8-ce8b-4dd3-99b9-abe7cacabea3 2 2024-09-22T03:01:34.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:34.534+0000 7f0bbdf97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:34.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:34.543+0000 7f0bbdf97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:34.676 INFO:teuthology.orchestra.run.smithi039.stdout:2 2024-09-22T03:01:34.687 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T03:01:34.718 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:01:34.718 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:01:34.760 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:01:34.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T03:01:34.791 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:01:34.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:01:34.834 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:01:34.834 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T03:01:34.868 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:01:34.868 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:01:34.910 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:01:34.911 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:01:35.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:35.053+0000 7fa934cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:35.063+0000 7fa934cd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:35.152 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:35.152 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":6,"fsid":"df5703b9-69b9-43c4-b0b0-b568274428f6","created":"2024-09-22T03:01:32.774495+0000","modified":"2024-09-22T03:01:34.674342+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:01:34.674342+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1471c1f6-ee1d-4ea6-ae1f-1dd37999c2e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f6b2e13d-3702-4ba6-b46b-cd4963228832","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2f051db8-ce8b-4dd3-99b9-abe7cacabea3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:01:35.176 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T03:01:35.176 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T03:01:35.311 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:35.309+0000 7f5188227f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-22T03:01:35.378 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:35.375+0000 7f5188227f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-22T03:01:35.478 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:01:35.580 INFO:teuthology.misc.health.smithi039.stderr:2024-09-22T03:01:35.575+0000 7f37b1197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:35.584 INFO:teuthology.misc.health.smithi039.stderr:2024-09-22T03:01:35.582+0000 7f37b1197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:35.653 INFO:teuthology.misc.health.smithi039.stdout: 2024-09-22T03:01:35.653 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":6,"fsid":"df5703b9-69b9-43c4-b0b0-b568274428f6","created":"2024-09-22T03:01:32.774495+0000","modified":"2024-09-22T03:01:34.674342+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:01:34.674342+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1471c1f6-ee1d-4ea6-ae1f-1dd37999c2e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f6b2e13d-3702-4ba6-b46b-cd4963228832","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2f051db8-ce8b-4dd3-99b9-abe7cacabea3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:01:35.663 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-22T03:01:35.795 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:35.792+0000 7f5188227f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-22T03:01:35.891 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:35.889+0000 7f5188227f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-22T03:01:36.011 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.009+0000 7f5188227f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:01:36.063 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.061+0000 7f5188227f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-22T03:01:36.188 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.186+0000 7f5188227f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-22T03:01:36.247 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.245+0000 7f5188227f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-22T03:01:36.298 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.296+0000 7f5188227f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-22T03:01:36.422 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.420+0000 7f5188227f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-22T03:01:36.477 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.475+0000 7f5188227f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-22T03:01:36.533 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.531+0000 7f5188227f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-22T03:01:36.606 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.604+0000 7f5188227f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-22T03:01:36.654 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.652+0000 7f5188227f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-22T03:01:36.702 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:36.699+0000 7f5188227f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-22T03:01:37.604 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:37.601+0000 7f5188227f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-22T03:01:38.612 INFO:tasks.ceph.mgr.x.smithi039.stderr:2024-09-22T03:01:38.610+0000 7f5126fd2640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T03:01:41.966 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:01:42.078 INFO:teuthology.misc.health.smithi039.stderr:2024-09-22T03:01:42.073+0000 7faa25c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.081 INFO:teuthology.misc.health.smithi039.stderr:2024-09-22T03:01:42.079+0000 7faa25c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.270 INFO:teuthology.misc.health.smithi039.stdout: 2024-09-22T03:01:42.270 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":9,"fsid":"df5703b9-69b9-43c4-b0b0-b568274428f6","created":"2024-09-22T03:01:32.774495+0000","modified":"2024-09-22T03:01:41.680033+0000","last_up_change":"2024-09-22T03:01:39.668223+0000","last_in_change":"2024-09-22T03:01:34.674342+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:01:41.669564+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1471c1f6-ee1d-4ea6-ae1f-1dd37999c2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2356902669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":2356902669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2356902669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":2356902669}]},"public_addr":"172.21.15.39:6806/2356902669","cluster_addr":"172.21.15.39:6803/2356902669","heartbeat_back_addr":"172.21.15.39:6812/2356902669","heartbeat_front_addr":"172.21.15.39:6811/2356902669","state":["exists","up"]},{"osd":1,"uuid":"f6b2e13d-3702-4ba6-b46b-cd4963228832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3532051251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3532051251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3532051251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3532051251}]},"public_addr":"172.21.15.39:6805/3532051251","cluster_addr":"172.21.15.39:6804/3532051251","heartbeat_back_addr":"172.21.15.39:6809/3532051251","heartbeat_front_addr":"172.21.15.39:6808/3532051251","state":["exists","up"]},{"osd":2,"uuid":"2f051db8-ce8b-4dd3-99b9-abe7cacabea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1848712837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":1848712837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1848712837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":1848712837}]},"public_addr":"172.21.15.39:6802/1848712837","cluster_addr":"172.21.15.39:6807/1848712837","heartbeat_back_addr":"172.21.15.39:6810/1848712837","heartbeat_front_addr":"172.21.15.39:6813/1848712837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:01:42.281 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-22T03:01:42.282 INFO:tasks.ceph:Creating RBD pool 2024-09-22T03:01:42.282 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T03:01:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:42.380+0000 7f4664e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:42.389+0000 7f4664e2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.691 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'rbd' created 2024-09-22T03:01:42.708 DEBUG:teuthology.orchestra.run.smithi039:> rbd --cluster ceph pool init rbd 2024-09-22T03:01:42.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:42.769+0000 7f2978440d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:42.769+0000 7f2978440d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:42.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:42.774+0000 7f2978440d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:45.778 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T03:01:45.778 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T03:01:45.778 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T03:01:45.779 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T03:01:45.779 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T03:01:45.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:45.886+0000 7f43a9797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:45.893+0000 7f43a9797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.135 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:46.154 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":3589726049},{"type":"v1","addr":"172.21.15.39:6801","nonce":3589726049}]},"active_addr":"172.21.15.39:6801/3589726049","active_change":"2024-09-22T03:01:37.604797+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":72520862}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":1830987718}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":3490990555}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":4280573770}]}]} 2024-09-22T03:01:46.156 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T03:01:46.156 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:01:46.157 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:01:46.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.266+0000 7f0a72b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.272+0000 7f0a72b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.463 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:46.463 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":13,"fsid":"df5703b9-69b9-43c4-b0b0-b568274428f6","created":"2024-09-22T03:01:32.774495+0000","modified":"2024-09-22T03:01:45.727617+0000","last_up_change":"2024-09-22T03:01:39.668223+0000","last_in_change":"2024-09-22T03:01:34.674342+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:01:41.669564+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:01:42.607262+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1471c1f6-ee1d-4ea6-ae1f-1dd37999c2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2356902669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":2356902669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2356902669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":2356902669}]},"public_addr":"172.21.15.39:6806/2356902669","cluster_addr":"172.21.15.39:6803/2356902669","heartbeat_back_addr":"172.21.15.39:6812/2356902669","heartbeat_front_addr":"172.21.15.39:6811/2356902669","state":["exists","up"]},{"osd":1,"uuid":"f6b2e13d-3702-4ba6-b46b-cd4963228832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3532051251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3532051251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3532051251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3532051251}]},"public_addr":"172.21.15.39:6805/3532051251","cluster_addr":"172.21.15.39:6804/3532051251","heartbeat_back_addr":"172.21.15.39:6809/3532051251","heartbeat_front_addr":"172.21.15.39:6808/3532051251","state":["exists","up"]},{"osd":2,"uuid":"2f051db8-ce8b-4dd3-99b9-abe7cacabea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1848712837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":1848712837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1848712837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":1848712837}]},"public_addr":"172.21.15.39:6802/1848712837","cluster_addr":"172.21.15.39:6807/1848712837","heartbeat_back_addr":"172.21.15.39:6810/1848712837","heartbeat_front_addr":"172.21.15.39:6813/1848712837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:01:46.479 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:01:46.480 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:01:46.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.581+0000 7f410a157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.587+0000 7f410a157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.776 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:46.777 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":13,"fsid":"df5703b9-69b9-43c4-b0b0-b568274428f6","created":"2024-09-22T03:01:32.774495+0000","modified":"2024-09-22T03:01:45.727617+0000","last_up_change":"2024-09-22T03:01:39.668223+0000","last_in_change":"2024-09-22T03:01:34.674342+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:01:41.669564+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:01:42.607262+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1471c1f6-ee1d-4ea6-ae1f-1dd37999c2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2356902669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":2356902669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2356902669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":2356902669}]},"public_addr":"172.21.15.39:6806/2356902669","cluster_addr":"172.21.15.39:6803/2356902669","heartbeat_back_addr":"172.21.15.39:6812/2356902669","heartbeat_front_addr":"172.21.15.39:6811/2356902669","state":["exists","up"]},{"osd":1,"uuid":"f6b2e13d-3702-4ba6-b46b-cd4963228832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3532051251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3532051251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3532051251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3532051251}]},"public_addr":"172.21.15.39:6805/3532051251","cluster_addr":"172.21.15.39:6804/3532051251","heartbeat_back_addr":"172.21.15.39:6809/3532051251","heartbeat_front_addr":"172.21.15.39:6808/3532051251","state":["exists","up"]},{"osd":2,"uuid":"2f051db8-ce8b-4dd3-99b9-abe7cacabea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1848712837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":1848712837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1848712837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":1848712837}]},"public_addr":"172.21.15.39:6802/1848712837","cluster_addr":"172.21.15.39:6807/1848712837","heartbeat_back_addr":"172.21.15.39:6810/1848712837","heartbeat_front_addr":"172.21.15.39:6813/1848712837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:01:46.794 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:01:46.795 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:01:46.795 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:01:46.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.911+0000 7f410874e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.911+0000 7f5145997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.912+0000 7fdef342c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.919+0000 7f410874e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.919+0000 7f5145997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:46.920+0000 7fdef342c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:46.948 INFO:teuthology.orchestra.run.smithi039.stdout:34359738375 2024-09-22T03:01:46.948 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:01:46.955 INFO:teuthology.orchestra.run.smithi039.stdout:34359738375 2024-09-22T03:01:46.955 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:01:46.955 INFO:teuthology.orchestra.run.smithi039.stdout:34359738375 2024-09-22T03:01:46.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:01:47.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:47.061+0000 7f1c3164f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:47.068+0000 7f1c3164f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:47.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:47.066+0000 7f3af086c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:47.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:47.067+0000 7f4fbd818640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:47.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:47.079+0000 7f4fbd818640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:47.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:47.079+0000 7f3af086c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:47.263 INFO:teuthology.orchestra.run.smithi039.stdout:34359738374 2024-09-22T03:01:47.274 INFO:teuthology.orchestra.run.smithi039.stdout:34359738374 2024-09-22T03:01:47.276 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738374 for osd.2 2024-09-22T03:01:47.281 INFO:teuthology.orchestra.run.smithi039.stdout:34359738374 2024-09-22T03:01:47.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738374 for osd.1 2024-09-22T03:01:47.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738374 for osd.0 2024-09-22T03:01:48.277 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:01:48.289 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:01:48.296 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:01:48.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:48.396+0000 7f533a2c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:48.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:48.407+0000 7f533a2c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:48.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:48.436+0000 7fde47b17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:48.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:48.439+0000 7f8af022c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:48.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:48.443+0000 7fde47b17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:48.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:48.447+0000 7f8af022c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:48.613 INFO:teuthology.orchestra.run.smithi039.stdout:34359738375 2024-09-22T03:01:48.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.2 2024-09-22T03:01:48.627 DEBUG:teuthology.parallel:result is None 2024-09-22T03:01:48.641 INFO:teuthology.orchestra.run.smithi039.stdout:34359738375 2024-09-22T03:01:48.652 INFO:teuthology.orchestra.run.smithi039.stdout:34359738375 2024-09-22T03:01:48.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2024-09-22T03:01:48.654 DEBUG:teuthology.parallel:result is None 2024-09-22T03:01:48.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.0 2024-09-22T03:01:48.666 DEBUG:teuthology.parallel:result is None 2024-09-22T03:01:48.666 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:01:48.666 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:01:48.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:48.809+0000 7eff3c597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:48.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:48.814+0000 7eff3c597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:49.000 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-22T03:01:49.013 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T03:01:47.612888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3324,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889860,"statfs":{"total":160658620416,"available":160655216640,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.906937"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.756756+0000","last_change":"2024-09-22T03:01:45.758866+0000","last_active":"2024-09-22T03:01:45.756756+0000","last_peered":"2024-09-22T03:01:45.756756+0000","last_clean":"2024-09-22T03:01:45.756756+0000","last_became_active":"2024-09-22T03:01:43.765548+0000","last_became_peered":"2024-09-22T03:01:43.765548+0000","last_unstale":"2024-09-22T03:01:45.756756+0000","last_undegraded":"2024-09-22T03:01:45.756756+0000","last_fullsized":"2024-09-22T03:01:45.756756+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.760093+0000","last_change":"2024-09-22T03:01:45.766286+0000","last_active":"2024-09-22T03:01:45.760093+0000","last_peered":"2024-09-22T03:01:45.760093+0000","last_clean":"2024-09-22T03:01:45.760093+0000","last_became_active":"2024-09-22T03:01:43.786726+0000","last_became_peered":"2024-09-22T03:01:43.786726+0000","last_unstale":"2024-09-22T03:01:45.760093+0000","last_undegraded":"2024-09-22T03:01:45.760093+0000","last_fullsized":"2024-09-22T03:01:45.760093+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.761133+0000","last_change":"2024-09-22T03:01:45.767686+0000","last_active":"2024-09-22T03:01:45.761133+0000","last_peered":"2024-09-22T03:01:45.761133+0000","last_clean":"2024-09-22T03:01:45.761133+0000","last_became_active":"2024-09-22T03:01:43.774106+0000","last_became_peered":"2024-09-22T03:01:43.774106+0000","last_unstale":"2024-09-22T03:01:45.761133+0000","last_undegraded":"2024-09-22T03:01:45.761133+0000","last_fullsized":"2024-09-22T03:01:45.761133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.760587+0000","last_change":"2024-09-22T03:01:45.767534+0000","last_active":"2024-09-22T03:01:45.760587+0000","last_peered":"2024-09-22T03:01:45.760587+0000","last_clean":"2024-09-22T03:01:45.760587+0000","last_became_active":"2024-09-22T03:01:43.777551+0000","last_became_peered":"2024-09-22T03:01:43.777551+0000","last_unstale":"2024-09-22T03:01:45.760587+0000","last_undegraded":"2024-09-22T03:01:45.760587+0000","last_fullsized":"2024-09-22T03:01:45.760587+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"12'1","reported_seq":17,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.760741+0000","last_change":"2024-09-22T03:01:45.760741+0000","last_active":"2024-09-22T03:01:45.760741+0000","last_peered":"2024-09-22T03:01:45.760741+0000","last_clean":"2024-09-22T03:01:45.760741+0000","last_became_active":"2024-09-22T03:01:43.844167+0000","last_became_peered":"2024-09-22T03:01:43.844167+0000","last_unstale":"2024-09-22T03:01:45.760741+0000","last_undegraded":"2024-09-22T03:01:45.760741+0000","last_fullsized":"2024-09-22T03:01:45.760741+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.759700+0000","last_change":"2024-09-22T03:01:45.762398+0000","last_active":"2024-09-22T03:01:45.759700+0000","last_peered":"2024-09-22T03:01:45.759700+0000","last_clean":"2024-09-22T03:01:45.759700+0000","last_became_active":"2024-09-22T03:01:43.814273+0000","last_became_peered":"2024-09-22T03:01:43.814273+0000","last_unstale":"2024-09-22T03:01:45.759700+0000","last_undegraded":"2024-09-22T03:01:45.759700+0000","last_fullsized":"2024-09-22T03:01:45.759700+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.755663+0000","last_change":"2024-09-22T03:01:45.758763+0000","last_active":"2024-09-22T03:01:45.755663+0000","last_peered":"2024-09-22T03:01:45.755663+0000","last_clean":"2024-09-22T03:01:45.755663+0000","last_became_active":"2024-09-22T03:01:43.789338+0000","last_became_peered":"2024-09-22T03:01:43.789338+0000","last_unstale":"2024-09-22T03:01:45.755663+0000","last_undegraded":"2024-09-22T03:01:45.755663+0000","last_fullsized":"2024-09-22T03:01:45.755663+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":16,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.759310+0000","last_change":"2024-09-22T03:01:45.767160+0000","last_active":"2024-09-22T03:01:45.759310+0000","last_peered":"2024-09-22T03:01:45.759310+0000","last_clean":"2024-09-22T03:01:45.759310+0000","last_became_active":"2024-09-22T03:01:43.756369+0000","last_became_peered":"2024-09-22T03:01:43.756369+0000","last_unstale":"2024-09-22T03:01:45.759310+0000","last_undegraded":"2024-09-22T03:01:45.759310+0000","last_fullsized":"2024-09-22T03:01:45.759310+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.754802+0000","last_change":"2024-09-22T03:01:42.751062+0000","last_active":"2024-09-22T03:01:45.754802+0000","last_peered":"2024-09-22T03:01:45.754802+0000","last_clean":"2024-09-22T03:01:45.754802+0000","last_became_active":"2024-09-22T03:01:42.736481+0000","last_became_peered":"2024-09-22T03:01:42.736481+0000","last_unstale":"2024-09-22T03:01:45.754802+0000","last_undegraded":"2024-09-22T03:01:45.754802+0000","last_fullsized":"2024-09-22T03:01:45.754802+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_clean_scrub_stamp":"2024-09-22T03:01:41.680033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655216640,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":420,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297308,"statfs":{"total":53552873472,"available":53552443392,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1580,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296148,"statfs":{"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1324,"kb_used_data":1324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296404,"statfs":{"total":53552873472,"available":53551517696,"internally_reserved":0,"allocated":1355776,"data_stored":1355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551517696,"internally_reserved":0,"allocated":1355776,"data_stored":1355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552443392,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:01:49.014 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:01:49.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:49.115+0000 7fde54d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:49.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:49.122+0000 7fde54d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:49.306 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:49.306 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-22T03:01:49.317 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T03:01:47.612888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3324,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889860,"statfs":{"total":160658620416,"available":160655216640,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.906937"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.756756+0000","last_change":"2024-09-22T03:01:45.758866+0000","last_active":"2024-09-22T03:01:45.756756+0000","last_peered":"2024-09-22T03:01:45.756756+0000","last_clean":"2024-09-22T03:01:45.756756+0000","last_became_active":"2024-09-22T03:01:43.765548+0000","last_became_peered":"2024-09-22T03:01:43.765548+0000","last_unstale":"2024-09-22T03:01:45.756756+0000","last_undegraded":"2024-09-22T03:01:45.756756+0000","last_fullsized":"2024-09-22T03:01:45.756756+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.760093+0000","last_change":"2024-09-22T03:01:45.766286+0000","last_active":"2024-09-22T03:01:45.760093+0000","last_peered":"2024-09-22T03:01:45.760093+0000","last_clean":"2024-09-22T03:01:45.760093+0000","last_became_active":"2024-09-22T03:01:43.786726+0000","last_became_peered":"2024-09-22T03:01:43.786726+0000","last_unstale":"2024-09-22T03:01:45.760093+0000","last_undegraded":"2024-09-22T03:01:45.760093+0000","last_fullsized":"2024-09-22T03:01:45.760093+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.761133+0000","last_change":"2024-09-22T03:01:45.767686+0000","last_active":"2024-09-22T03:01:45.761133+0000","last_peered":"2024-09-22T03:01:45.761133+0000","last_clean":"2024-09-22T03:01:45.761133+0000","last_became_active":"2024-09-22T03:01:43.774106+0000","last_became_peered":"2024-09-22T03:01:43.774106+0000","last_unstale":"2024-09-22T03:01:45.761133+0000","last_undegraded":"2024-09-22T03:01:45.761133+0000","last_fullsized":"2024-09-22T03:01:45.761133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.760587+0000","last_change":"2024-09-22T03:01:45.767534+0000","last_active":"2024-09-22T03:01:45.760587+0000","last_peered":"2024-09-22T03:01:45.760587+0000","last_clean":"2024-09-22T03:01:45.760587+0000","last_became_active":"2024-09-22T03:01:43.777551+0000","last_became_peered":"2024-09-22T03:01:43.777551+0000","last_unstale":"2024-09-22T03:01:45.760587+0000","last_undegraded":"2024-09-22T03:01:45.760587+0000","last_fullsized":"2024-09-22T03:01:45.760587+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"12'1","reported_seq":17,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.760741+0000","last_change":"2024-09-22T03:01:45.760741+0000","last_active":"2024-09-22T03:01:45.760741+0000","last_peered":"2024-09-22T03:01:45.760741+0000","last_clean":"2024-09-22T03:01:45.760741+0000","last_became_active":"2024-09-22T03:01:43.844167+0000","last_became_peered":"2024-09-22T03:01:43.844167+0000","last_unstale":"2024-09-22T03:01:45.760741+0000","last_undegraded":"2024-09-22T03:01:45.760741+0000","last_fullsized":"2024-09-22T03:01:45.760741+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.759700+0000","last_change":"2024-09-22T03:01:45.762398+0000","last_active":"2024-09-22T03:01:45.759700+0000","last_peered":"2024-09-22T03:01:45.759700+0000","last_clean":"2024-09-22T03:01:45.759700+0000","last_became_active":"2024-09-22T03:01:43.814273+0000","last_became_peered":"2024-09-22T03:01:43.814273+0000","last_unstale":"2024-09-22T03:01:45.759700+0000","last_undegraded":"2024-09-22T03:01:45.759700+0000","last_fullsized":"2024-09-22T03:01:45.759700+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.755663+0000","last_change":"2024-09-22T03:01:45.758763+0000","last_active":"2024-09-22T03:01:45.755663+0000","last_peered":"2024-09-22T03:01:45.755663+0000","last_clean":"2024-09-22T03:01:45.755663+0000","last_became_active":"2024-09-22T03:01:43.789338+0000","last_became_peered":"2024-09-22T03:01:43.789338+0000","last_unstale":"2024-09-22T03:01:45.755663+0000","last_undegraded":"2024-09-22T03:01:45.755663+0000","last_fullsized":"2024-09-22T03:01:45.755663+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":16,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.759310+0000","last_change":"2024-09-22T03:01:45.767160+0000","last_active":"2024-09-22T03:01:45.759310+0000","last_peered":"2024-09-22T03:01:45.759310+0000","last_clean":"2024-09-22T03:01:45.759310+0000","last_became_active":"2024-09-22T03:01:43.756369+0000","last_became_peered":"2024-09-22T03:01:43.756369+0000","last_unstale":"2024-09-22T03:01:45.759310+0000","last_undegraded":"2024-09-22T03:01:45.759310+0000","last_fullsized":"2024-09-22T03:01:45.759310+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:01:45.754802+0000","last_change":"2024-09-22T03:01:42.751062+0000","last_active":"2024-09-22T03:01:45.754802+0000","last_peered":"2024-09-22T03:01:45.754802+0000","last_clean":"2024-09-22T03:01:45.754802+0000","last_became_active":"2024-09-22T03:01:42.736481+0000","last_became_peered":"2024-09-22T03:01:42.736481+0000","last_unstale":"2024-09-22T03:01:45.754802+0000","last_undegraded":"2024-09-22T03:01:45.754802+0000","last_fullsized":"2024-09-22T03:01:45.754802+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_clean_scrub_stamp":"2024-09-22T03:01:41.680033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655216640,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":420,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297308,"statfs":{"total":53552873472,"available":53552443392,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1580,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296148,"statfs":{"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1324,"kb_used_data":1324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296404,"statfs":{"total":53552873472,"available":53551517696,"internally_reserved":0,"allocated":1355776,"data_stored":1355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551517696,"internally_reserved":0,"allocated":1355776,"data_stored":1355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552443392,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:01:49.318 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:01:49.318 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T03:01:49.318 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T03:01:49.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T03:01:49.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:49.421+0000 7f12be24f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:49.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:01:49.426+0000 7f12be24f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:01:49.652 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:01:49.652 INFO:teuthology.orchestra.run.smithi039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T03:01:49.665 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T03:01:49.665 INFO:teuthology.run_tasks:Running task workunit... 2024-09-22T03:01:49.684 INFO:tasks.workunit:Pulling workunits from ref 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:01:49.684 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-22T03:01:49.684 DEBUG:teuthology.orchestra.run.smithi039:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-22T03:01:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:01:49.701 INFO:teuthology.orchestra.run.smithi039.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-22T03:01:49.701 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-22T03:01:49.757 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-22T03:01:49.757 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-22T03:01:49.813 INFO:tasks.workunit:timeout=3h 2024-09-22T03:01:49.813 INFO:tasks.workunit:cleanup=True 2024-09-22T03:01:49.814 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:01:49.869 INFO:tasks.workunit.client.0.smithi039.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-22T03:02:46.294 INFO:tasks.workunit.client.0.smithi039.stderr:Updating files: 94% (12158/12896) Updating files: 95% (12252/12896) Updating files: 96% (12381/12896) Updating files: 97% (12510/12896) Updating files: 98% (12639/12896) Updating files: 99% (12768/12896) Updating files: 100% (12896/12896) Updating files: 100% (12896/12896), done. 2024-09-22T03:02:46.905 INFO:tasks.workunit.client.0.smithi039.stderr:Note: switching to '4e45e6b490e6cbe082cdf155fea780438ec53fc7'. 2024-09-22T03:02:46.905 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-22T03:02:46.905 INFO:tasks.workunit.client.0.smithi039.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-22T03:02:46.905 INFO:tasks.workunit.client.0.smithi039.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-22T03:02:46.905 INFO:tasks.workunit.client.0.smithi039.stderr:state without impacting any branches by switching back to a branch. 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr: git switch -c 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr:Or undo this operation with: 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr: git switch - 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-22T03:02:46.906 INFO:tasks.workunit.client.0.smithi039.stderr:HEAD is now at 4e45e6b490e Merge pull request #59907 from zdover23/wip-doc-2024-09-21-backport-59904-to-squid 2024-09-22T03:02:46.911 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-22T03:02:46.974 INFO:tasks.workunit.client.0.smithi039.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-22T03:02:46.976 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-22T03:02:46.976 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-22T03:02:47.631 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-22T03:02:47.697 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-22T03:02:47.739 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-22T03:02:47.740 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-22T03:02:47.741 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-22T03:02:47.783 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-22T03:02:47.786 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-22T03:02:47.786 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-22T03:02:47.843 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2024-09-22T03:02:47.844 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2024-09-22T03:02:47.844 DEBUG:teuthology.orchestra.run.smithi039:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-09-22T03:02:47.941 INFO:tasks.workunit.client.0.smithi039.stdout:seed 29677 2024-09-22T03:02:47.942 INFO:tasks.workunit.client.0.smithi039.stdout:Note: Google Test filter = TestLibRBD.* 2024-09-22T03:02:47.942 INFO:tasks.workunit.client.0.smithi039.stdout:[==========] Running 170 tests from 1 test suite. 2024-09-22T03:02:47.942 INFO:tasks.workunit.client.0.smithi039.stdout:[----------] Global test environment set-up. 2024-09-22T03:02:47.942 INFO:tasks.workunit.client.0.smithi039.stdout:[----------] 170 tests from TestLibRBD 2024-09-22T03:02:47.979 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-09-22T03:02:49.916 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.017 INFO:tasks.workunit.client.0.smithi039.stdout:image has size 2097152 and order 22 2024-09-22T03:02:52.017 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.CreateAndStat (4038 ms) 2024-09-22T03:02:52.017 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-09-22T03:02:52.017 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:780: Skipped 2024-09-22T03:02:52.018 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:02:52.018 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool (0 ms) 2024-09-22T03:02:52.018 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-09-22T03:02:52.018 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.081 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.CreateAndStatPP (64 ms) 2024-09-22T03:02:52.081 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.GetId 2024-09-22T03:02:52.081 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.146 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.GetId (65 ms) 2024-09-22T03:02:52.146 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-09-22T03:02:52.146 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.211 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.GetIdPP (65 ms) 2024-09-22T03:02:52.211 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-09-22T03:02:52.211 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.269 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (58 ms) 2024-09-22T03:02:52.269 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-09-22T03:02:52.269 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.327 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (58 ms) 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:939: Skipped 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp (0 ms) 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:962: Skipped 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP (0 ms) 2024-09-22T03:02:52.328 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.OpenAio 2024-09-22T03:02:52.329 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.385 INFO:tasks.workunit.client.0.smithi039.stdout:image has size 2097152 and order 22 2024-09-22T03:02:52.385 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.OpenAio (58 ms) 2024-09-22T03:02:52.385 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-09-22T03:02:52.393 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.OpenAioFail (8 ms) 2024-09-22T03:02:52.393 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-09-22T03:02:52.393 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.485 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.OpenAioPP (92 ms) 2024-09-22T03:02:52.485 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-09-22T03:02:52.494 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.OpenAioFailPP (9 ms) 2024-09-22T03:02:52.494 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-09-22T03:02:52.494 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.640 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.ResizeAndStat (146 ms) 2024-09-22T03:02:52.640 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-09-22T03:02:52.640 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.748 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (109 ms) 2024-09-22T03:02:52.748 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-09-22T03:02:52.748 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:52.878 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (129 ms) 2024-09-22T03:02:52.878 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-09-22T03:02:52.878 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:53.015 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (137 ms) 2024-09-22T03:02:53.015 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-09-22T03:02:55.117 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:57.027 INFO:tasks.workunit.client.0.smithi039.stdout:image: image15 2024-09-22T03:02:57.027 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image15 2024-09-22T03:02:57.027 INFO:tasks.workunit.client.0.smithi039.stdout:found image15 2024-09-22T03:02:57.027 INFO:tasks.workunit.client.0.smithi039.stdout:erased image15 2024-09-22T03:02:57.027 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:02:57.181 INFO:tasks.workunit.client.0.smithi039.stdout:image: image15 2024-09-22T03:02:57.181 INFO:tasks.workunit.client.0.smithi039.stdout:image: image16 2024-09-22T03:02:57.181 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image15 2024-09-22T03:02:57.181 INFO:tasks.workunit.client.0.smithi039.stdout:found image15 2024-09-22T03:02:57.181 INFO:tasks.workunit.client.0.smithi039.stdout:erased image15 2024-09-22T03:02:57.181 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image16 2024-09-22T03:02:57.181 INFO:tasks.workunit.client.0.smithi039.stdout:found image16 2024-09-22T03:02:57.181 INFO:tasks.workunit.client.0.smithi039.stdout:erased image16 2024-09-22T03:02:57.182 INFO:tasks.workunit.client.0.smithi039.stdout:image: image16 2024-09-22T03:02:57.182 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image16 2024-09-22T03:02:57.182 INFO:tasks.workunit.client.0.smithi039.stdout:found image16 2024-09-22T03:02:57.182 INFO:tasks.workunit.client.0.smithi039.stdout:erased image16 2024-09-22T03:02:57.182 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (4166 ms) 2024-09-22T03:02:57.182 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-09-22T03:02:58.955 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:01.065 INFO:tasks.workunit.client.0.smithi039.stdout:num images is: 1 2024-09-22T03:03:01.065 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:01.065 INFO:tasks.workunit.client.0.smithi039.stdout:image: image17 2024-09-22T03:03:01.065 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image17 2024-09-22T03:03:01.065 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:01.103 INFO:tasks.workunit.client.0.smithi039.stdout:num images is: 2 2024-09-22T03:03:01.103 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 2 2024-09-22T03:03:01.103 INFO:tasks.workunit.client.0.smithi039.stdout:image: image17 2024-09-22T03:03:01.103 INFO:tasks.workunit.client.0.smithi039.stdout:image: image18 2024-09-22T03:03:01.103 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image17 2024-09-22T03:03:01.103 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image18 2024-09-22T03:03:01.180 INFO:tasks.workunit.client.0.smithi039.stdout:num images is: 1 2024-09-22T03:03:01.181 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:01.181 INFO:tasks.workunit.client.0.smithi039.stdout:image: image18 2024-09-22T03:03:01.181 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image18 2024-09-22T03:03:01.181 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (4000 ms) 2024-09-22T03:03:01.181 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCopy 2024-09-22T03:03:02.983 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:08.990 INFO:tasks.workunit.client.0.smithi039.stdout:image: image19 2024-09-22T03:03:08.990 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image19 2024-09-22T03:03:08.990 INFO:tasks.workunit.client.0.smithi039.stdout:found image19 2024-09-22T03:03:08.990 INFO:tasks.workunit.client.0.smithi039.stdout:erased image19 2024-09-22T03:03:08.990 INFO:tasks.workunit.client.0.smithi039.stdout:image: image19 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:image: image20 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image19 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:found image19 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:erased image19 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image20 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:found image20 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:erased image20 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:0.00% done 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:100.00% done 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:image: image19 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:image: image20 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:image: image21 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image19 2024-09-22T03:03:08.991 INFO:tasks.workunit.client.0.smithi039.stdout:found image19 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:erased image19 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image20 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:found image20 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:erased image20 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image21 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:found image21 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:erased image21 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCopy (7809 ms) 2024-09-22T03:03:08.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-09-22T03:03:11.026 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:14.739 INFO:tasks.workunit.client.0.smithi039.stdout:num images is: 1 2024-09-22T03:03:14.739 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:14.739 INFO:tasks.workunit.client.0.smithi039.stdout:image: image22 2024-09-22T03:03:14.739 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image22 2024-09-22T03:03:16.303 INFO:tasks.workunit.client.0.smithi039.stdout:num images is: 2 2024-09-22T03:03:16.304 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 2 2024-09-22T03:03:16.304 INFO:tasks.workunit.client.0.smithi039.stdout:image: image22 2024-09-22T03:03:16.304 INFO:tasks.workunit.client.0.smithi039.stdout:image: image23 2024-09-22T03:03:16.304 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image22 2024-09-22T03:03:16.304 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image23 2024-09-22T03:03:16.744 INFO:tasks.workunit.client.0.smithi039.stdout:0.00% done 2024-09-22T03:03:16.744 INFO:tasks.workunit.client.0.smithi039.stdout:100.00% done 2024-09-22T03:03:16.744 INFO:tasks.workunit.client.0.smithi039.stdout:num images is: 3 2024-09-22T03:03:16.744 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 3 2024-09-22T03:03:16.744 INFO:tasks.workunit.client.0.smithi039.stdout:image: image22 2024-09-22T03:03:16.744 INFO:tasks.workunit.client.0.smithi039.stdout:image: image23 2024-09-22T03:03:16.745 INFO:tasks.workunit.client.0.smithi039.stdout:image: image24 2024-09-22T03:03:16.745 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image22 2024-09-22T03:03:16.745 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image23 2024-09-22T03:03:16.745 INFO:tasks.workunit.client.0.smithi039.stdout:expected = image24 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCopyPP (8076 ms) 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:1578: Skipped 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy (0 ms) 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:1745: Skipped 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP (0 ms) 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-09-22T03:03:17.067 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:21.183 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 1 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap1 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:found snap1 with size 2097152 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 2 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 2 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap1 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap2 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:found snap1 with size 2097152 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:found snap2 with size 4194304 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 1 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap2 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:found snap2 with size 4194304 2024-09-22T03:03:21.184 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 0 2024-09-22T03:03:21.185 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 0 2024-09-22T03:03:21.185 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (4117 ms) 2024-09-22T03:03:21.185 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-09-22T03:03:21.185 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:1890: Skipped 2024-09-22T03:03:21.185 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:03:21.185 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp (0 ms) 2024-09-22T03:03:21.185 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-09-22T03:03:21.185 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:22.193 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 1 2024-09-22T03:03:22.193 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:22.193 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap1 2024-09-22T03:03:22.193 INFO:tasks.workunit.client.0.smithi039.stdout:found snap1 with size 2097152 2024-09-22T03:03:23.207 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 2 2024-09-22T03:03:23.207 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 2 2024-09-22T03:03:23.207 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap1 2024-09-22T03:03:23.207 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap2 2024-09-22T03:03:23.207 INFO:tasks.workunit.client.0.smithi039.stdout:found snap1 with size 2097152 2024-09-22T03:03:23.207 INFO:tasks.workunit.client.0.smithi039.stdout:found snap2 with size 4194304 2024-09-22T03:03:24.210 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 1 2024-09-22T03:03:24.211 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:24.211 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap2 2024-09-22T03:03:24.211 INFO:tasks.workunit.client.0.smithi039.stdout:found snap2 with size 4194304 2024-09-22T03:03:25.212 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 0 2024-09-22T03:03:25.212 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 0 2024-09-22T03:03:25.222 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4039 ms) 2024-09-22T03:03:25.223 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-09-22T03:03:25.223 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:31.269 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 0 2024-09-22T03:03:31.270 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 0 2024-09-22T03:03:31.278 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6056 ms) 2024-09-22T03:03:31.279 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-09-22T03:03:31.279 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:32.255 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 1 2024-09-22T03:03:32.255 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:32.255 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap1 2024-09-22T03:03:32.255 INFO:tasks.workunit.client.0.smithi039.stdout:found snap1 with size 2097152 2024-09-22T03:03:33.256 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 2 2024-09-22T03:03:33.256 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 2 2024-09-22T03:03:33.256 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap1 2024-09-22T03:03:33.256 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap2 2024-09-22T03:03:33.256 INFO:tasks.workunit.client.0.smithi039.stdout:found snap1 with size 2097152 2024-09-22T03:03:33.256 INFO:tasks.workunit.client.0.smithi039.stdout:found snap2 with size 4194304 2024-09-22T03:03:33.283 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 2 2024-09-22T03:03:33.283 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 2 2024-09-22T03:03:33.283 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap1-rename 2024-09-22T03:03:33.283 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap2 2024-09-22T03:03:33.284 INFO:tasks.workunit.client.0.smithi039.stdout:found snap1-rename with size 2097152 2024-09-22T03:03:33.284 INFO:tasks.workunit.client.0.smithi039.stdout:found snap2 with size 4194304 2024-09-22T03:03:34.302 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 1 2024-09-22T03:03:34.303 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:03:34.303 INFO:tasks.workunit.client.0.smithi039.stdout:snap: snap2-rename 2024-09-22T03:03:34.303 INFO:tasks.workunit.client.0.smithi039.stdout:found snap2-rename with size 4194304 2024-09-22T03:03:35.308 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 0 2024-09-22T03:03:35.308 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 0 2024-09-22T03:03:35.319 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (4040 ms) 2024-09-22T03:03:35.319 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-09-22T03:03:37.184 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:37.184 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:37.184 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:37.184 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:03:39.704 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (4385 ms) 2024-09-22T03:03:39.704 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-09-22T03:03:44.382 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:04:03.237 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.RemoveFullTry (23533 ms) 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2227: Skipped 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool (0 ms) 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2247: Skipped 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespace (0 ms) 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2269: Skipped 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespaceDataPool (0 ms) 2024-09-22T03:04:03.238 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestIO 2024-09-22T03:04:03.239 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.537 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.538 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:17.539 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.540 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:17.541 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:17.542 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:17.543 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:discard: 512~512 = 512 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:aio discard: 1536~512 = 0 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:04:17.544 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.545 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:04:17.546 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.547 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.548 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:17.550 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.551 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.552 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:17.552 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:17.552 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:17.552 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:17.552 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:17.552 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:17.552 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:17.552 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.807 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.807 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.807 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.807 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.807 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.807 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.807 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.807 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.808 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.809 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.810 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.811 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.812 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.813 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:29.814 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestIO (26570 ms) 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-09-22T03:04:29.815 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:04:50.609 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:50.609 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:50.609 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:50.609 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:50.609 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:04:50.609 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:50.610 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:50.611 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:50.612 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:04:50.613 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.614 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.615 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:discard: 512~512 = 512 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:aio discard: 1536~512 = 0 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.616 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.617 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.618 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.619 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.620 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.621 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.622 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.623 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:04:50.624 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:04:50.625 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:04:50.625 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.939 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.939 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.939 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.939 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.940 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.941 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.942 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.943 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.944 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.945 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.946 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:05:09.947 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:09.948 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:05:09.948 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 4 2024-09-22T03:05:09.948 INFO:tasks.workunit.client.0.smithi039.stdout:read: 4 2024-09-22T03:05:09.948 INFO:tasks.workunit.client.0.smithi039.stdout:read: 4 2024-09-22T03:05:09.948 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (40132 ms) 2024-09-22T03:05:09.948 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-09-22T03:05:09.948 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:05:17.001 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:17.001 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:17.001 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:17.001 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:17.001 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:17.002 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:05:17.003 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: -84 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:495: Failure 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout:Expected equality of these values: 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout: len 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout: Which is: 512 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout: static_cast(written) 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout: Which is: 18446744073709551532 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:630: Failure 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout:Value of: passed 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout: Actual: false 2024-09-22T03:05:17.004 INFO:tasks.workunit.client.0.smithi039.stdout:Expected: true 2024-09-22T03:05:21.147 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 4 2024-09-22T03:05:21.147 INFO:tasks.workunit.client.0.smithi039.stdout:read: 4 2024-09-22T03:05:21.147 INFO:tasks.workunit.client.0.smithi039.stdout:read: 4 2024-09-22T03:05:21.147 INFO:tasks.workunit.client.0.smithi039.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 (11208 ms) 2024-09-22T03:05:21.147 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2471: Skipped 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption (0 ms) 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2659: Skipped 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (0 ms) 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2735: Skipped 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (0 ms) 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-09-22T03:05:21.148 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:05:25.563 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4417 ms) 2024-09-22T03:05:25.563 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-09-22T03:05:25.564 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:05:31.420 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (5857 ms) 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2905: Skipped 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2985: Skipped 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch (0 ms) 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-09-22T03:05:31.421 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:05:43.056 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.EncryptedResize (11635 ms) 2024-09-22T03:05:43.056 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-09-22T03:05:43.056 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:3202: Skipped 2024-09-22T03:05:43.057 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:05:43.057 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData (0 ms) 2024-09-22T03:05:43.057 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-09-22T03:05:43.057 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:05:56.894 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:56.894 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:56.894 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:56.894 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:56.895 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:05:56.896 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.897 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.898 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:05:56.899 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.900 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:discard: 512~512 = 512 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:aio discard: 1536~512 = 0 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:05:56.901 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.902 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.903 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:05:56.904 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.905 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.906 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.907 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.908 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:05:56.909 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.728 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.728 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.728 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.728 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.728 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.728 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.728 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.729 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.730 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.731 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.732 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.733 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.734 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.735 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:08.736 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (25672 ms) 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:3947: Skipped 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO (0 ms) 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-09-22T03:06:08.737 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:08.819 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (91 ms) 2024-09-22T03:06:08.819 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-09-22T03:06:08.819 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:08.910 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (91 ms) 2024-09-22T03:06:08.910 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-09-22T03:06:08.910 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:09.006 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (96 ms) 2024-09-22T03:06:09.006 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-09-22T03:06:09.006 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:09.105 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (99 ms) 2024-09-22T03:06:09.105 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-09-22T03:06:09.105 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:09.454 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (348 ms) 2024-09-22T03:06:09.454 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-09-22T03:06:09.454 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:09.747 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (294 ms) 2024-09-22T03:06:09.747 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-09-22T03:06:09.747 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:10.098 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (351 ms) 2024-09-22T03:06:10.098 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-09-22T03:06:10.098 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:10.360 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (262 ms) 2024-09-22T03:06:10.360 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-09-22T03:06:10.360 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:12.147 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (1787 ms) 2024-09-22T03:06:12.147 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-09-22T03:06:12.147 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:13.922 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (1775 ms) 2024-09-22T03:06:13.922 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-09-22T03:06:13.922 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:14.007 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (85 ms) 2024-09-22T03:06:14.007 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-09-22T03:06:14.007 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:14.105 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (98 ms) 2024-09-22T03:06:14.105 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-09-22T03:06:14.105 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:14.198 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (93 ms) 2024-09-22T03:06:14.198 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-09-22T03:06:14.198 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:14.297 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (100 ms) 2024-09-22T03:06:14.298 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-09-22T03:06:14.298 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:14.374 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.375 INFO:tasks.workunit.client.0.smithi039.stdout:aio discard: 0~1048576 = 0 2024-09-22T03:06:14.375 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.375 INFO:tasks.workunit.client.0.smithi039.stdout:aio discard: 0~4194304 = 0 2024-09-22T03:06:14.375 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.375 INFO:tasks.workunit.client.0.smithi039.stdout:aio discard: 3145728~1048576 = 0 2024-09-22T03:06:14.375 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (76 ms) 2024-09-22T03:06:14.375 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestFUA 2024-09-22T03:06:14.375 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:14.694 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.694 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.694 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.695 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.695 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.695 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.695 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.695 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.695 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.695 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.696 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.697 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.698 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestFUA (320 ms) 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-09-22T03:06:14.699 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:14.748 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.748 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.749 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.749 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.749 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:14.749 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.749 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.749 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.749 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.749 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.750 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.750 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.750 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.750 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.750 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.750 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.750 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.751 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.751 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.751 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.751 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.751 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.751 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.751 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.752 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.752 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.752 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:14.752 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:06:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:06:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:06:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:06:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:06:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:06:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:06:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:06:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:start compare and write 2024-09-22T03:06:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:compare and wrote: 512 2024-09-22T03:06:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:06:14.890 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.899 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.899 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:06:14.900 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.900 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:06:14.900 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.910 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.910 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:06:14.911 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.911 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:06:14.911 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.921 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.921 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:06:14.921 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.921 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:06:14.921 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.932 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.932 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:06:14.932 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.932 INFO:tasks.workunit.client.0.smithi039.stdout:started aio compare and write 2024-09-22T03:06:14.932 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:finished aio compare and write 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.959 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.961 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.961 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:14.961 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.962 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.962 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.965 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.965 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:14.965 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.965 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.965 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.968 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.968 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:14.968 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.968 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.968 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.971 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.971 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:14.971 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:14.972 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:14.972 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:14.972 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:14.972 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:14.972 INFO:tasks.workunit.client.0.smithi039.stdout:discard: 512~512 2024-09-22T03:06:14.972 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.138 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.139 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.140 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.141 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.142 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:26.143 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.144 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:26.254 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:26.254 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:26.254 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.254 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.254 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.254 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.254 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:26.337 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:26.337 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:26.337 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.338 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:26.636 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:26.636 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:26.636 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.636 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.637 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.637 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.637 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:26.850 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:26.851 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:26.851 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:26.851 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:26.851 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:26.851 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:26.851 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:27.260 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:27.261 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:27.261 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:27.261 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:27.261 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:27.261 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:27.261 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:27.549 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:27.549 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:27.549 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:27.549 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:27.549 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:27.549 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:27.549 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:27.550 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:27.550 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:27.550 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:27.550 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:27.550 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:27.550 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:28.157 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:28.158 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:28.158 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:28.158 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:28.158 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:28.158 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:28.158 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:28.557 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:28.558 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:28.558 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:28.558 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:28.558 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:28.558 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:28.558 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:29.252 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:29.252 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:29.252 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:29.252 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:29.252 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:29.252 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:29.252 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:29.730 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:29.730 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:29.730 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:29.731 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:30.632 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:30.632 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:30.632 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:30.632 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:30.633 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:30.633 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:30.633 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:31.287 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:31.287 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:31.288 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:31.288 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:31.288 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:31.288 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:31.288 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:32.315 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:32.316 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:32.316 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:32.316 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:32.316 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:32.316 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:32.316 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:32.977 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:32.977 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:32.977 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:32.977 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:32.977 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:32.977 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:32.977 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:32.977 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:32.978 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:32.978 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:32.978 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:32.978 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:32.978 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:34.223 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:34.223 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:34.223 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:34.223 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:34.223 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:34.223 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:34.223 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:35.017 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:35.017 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:35.017 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:35.017 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:35.017 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:35.017 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:35.017 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:36.333 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:36.333 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:36.333 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:36.333 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:36.333 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:36.333 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:36.333 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:37.195 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:37.195 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:37.195 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:37.195 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:37.195 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:37.195 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:37.195 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:37.195 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:37.196 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:37.196 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:37.196 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestIOPP (22501 ms) 2024-09-22T03:06:37.196 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-09-22T03:06:37.196 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:37.275 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (80 ms) 2024-09-22T03:06:37.275 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-09-22T03:06:37.275 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:37.354 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (79 ms) 2024-09-22T03:06:37.354 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-09-22T03:06:37.354 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:37.437 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (83 ms) 2024-09-22T03:06:37.437 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-09-22T03:06:37.437 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:37.531 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (94 ms) 2024-09-22T03:06:37.531 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-09-22T03:06:37.531 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:37.641 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (110 ms) 2024-09-22T03:06:37.641 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-09-22T03:06:37.641 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:37.732 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:37.753 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (112 ms) 2024-09-22T03:06:37.753 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-22T03:06:37.753 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:37.845 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (92 ms) 2024-09-22T03:06:37.845 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-22T03:06:37.845 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:37.921 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:37.940 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (95 ms) 2024-09-22T03:06:37.940 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-09-22T03:06:37.940 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:38.043 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (103 ms) 2024-09-22T03:06:38.043 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-09-22T03:06:38.043 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:38.129 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:38.148 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (104 ms) 2024-09-22T03:06:38.148 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-22T03:06:38.148 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:38.259 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (111 ms) 2024-09-22T03:06:38.259 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-22T03:06:38.259 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:38.354 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:38.377 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (118 ms) 2024-09-22T03:06:38.377 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-09-22T03:06:38.377 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:38.668 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (291 ms) 2024-09-22T03:06:38.668 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-09-22T03:06:38.668 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:38.853 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:38.990 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (322 ms) 2024-09-22T03:06:38.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-09-22T03:06:38.993 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:39.344 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (354 ms) 2024-09-22T03:06:39.344 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-09-22T03:06:39.344 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:39.528 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:39.638 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (295 ms) 2024-09-22T03:06:39.639 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-09-22T03:06:39.639 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:41.602 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (1963 ms) 2024-09-22T03:06:41.602 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-09-22T03:06:41.603 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:43.322 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:43.521 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (1919 ms) 2024-09-22T03:06:43.521 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-09-22T03:06:43.521 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:06:43.673 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:43.673 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:43.673 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:43.673 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:43.673 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:06:43.673 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.674 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:43.674 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:43.684 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:43.684 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:43.684 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.684 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:43.685 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:43.695 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:43.696 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:43.696 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.696 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:43.696 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:43.707 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:43.707 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:43.707 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.707 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:43.707 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:43.718 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:43.718 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:43.719 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.719 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:06:43.719 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:43.730 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:43.730 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:06:43.730 INFO:tasks.workunit.client.0.smithi039.stdout:read: 512 2024-09-22T03:06:43.730 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.730 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:43.730 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:43.733 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:43.733 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:43.733 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.733 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:43.733 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:43.736 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:43.736 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:43.736 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.736 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:43.736 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:43.739 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:43.739 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:43.739 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.739 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:43.739 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:43.742 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:43.742 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:43.742 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:43.742 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:06:43.742 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:06:54.887 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:06:54.887 INFO:tasks.workunit.client.0.smithi039.stdout:finished read 2024-09-22T03:06:54.887 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 0 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.888 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 16384 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 49152 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 65536 2024-09-22T03:06:54.889 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 98304 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.890 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 114688 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 147456 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.891 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 163840 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 196608 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 212992 2024-09-22T03:06:54.892 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:54.893 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:54.899 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:54.899 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:54.899 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:54.899 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:54.899 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:54.899 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:54.899 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:55.022 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:55.022 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:55.022 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:55.022 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:55.022 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:55.022 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:55.022 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:55.102 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:55.104 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:55.105 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:55.513 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:55.514 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:55.514 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:55.514 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:55.514 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:55.514 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:55.514 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:55.720 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:55.720 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:55.721 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:55.721 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:55.721 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:55.721 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:55.721 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:56.136 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:56.136 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:56.136 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:56.136 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:56.136 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:56.136 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:56.136 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:56.412 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:56.413 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:57.013 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:57.014 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:57.014 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:57.014 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:57.014 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:57.014 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:57.014 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:57.422 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:57.422 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:57.422 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:57.422 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:57.422 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:57.422 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:57.422 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:58.170 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:58.171 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:58.171 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:58.171 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:58.171 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:58.171 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:58.171 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:58.640 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:58.640 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:58.640 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:58.640 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:58.640 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:58.641 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:58.641 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:58.641 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:58.641 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:58.641 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:06:58.641 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:58.641 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:58.641 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:06:59.581 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:06:59.581 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:06:59.581 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:06:59.581 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:06:59.581 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:06:59.581 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:06:59.581 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:00.176 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:00.176 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:07:00.176 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:07:00.176 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:07:00.177 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:00.177 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:00.177 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:01.158 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:01.159 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:07:01.159 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:07:01.159 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:07:01.159 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:01.159 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:01.159 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:01.854 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:01.854 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:07:01.854 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:07:01.854 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:07:01.854 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:01.854 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:01.854 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:07:01.855 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:01.855 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:01.855 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:07:01.855 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:01.855 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:01.855 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:03.048 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:03.048 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:07:03.048 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:07:03.048 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:07:03.048 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:03.048 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:03.048 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:03.933 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:03.933 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:07:03.933 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:07:03.933 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:07:03.933 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:03.933 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:03.933 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:05.212 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:05.212 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:07:05.212 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:07:05.212 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:07:05.212 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:05.212 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:05.212 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:finished writesame 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:to verify the data 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:verified 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:started writesame 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:expected fail, finished writesame 2024-09-22T03:07:06.120 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (22599 ms) 2024-09-22T03:07:06.121 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-09-22T03:07:06.121 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 0 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 0 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 1 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:snap: orig 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:found orig with size 2097152 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:read: 80 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:write test data! 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 80 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 2 2024-09-22T03:07:08.146 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 2 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:snap: orig 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:snap: written 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:found orig with size 2097152 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:found written with size 2097152 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:read: 80 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:read: 80 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:read: 80 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:write to snapshot returned -30 2024-09-22T03:07:08.147 INFO:tasks.workunit.client.0.smithi039.stdout:Read-only file system 2024-09-22T03:07:08.225 INFO:tasks.workunit.client.0.smithi039.stdout:read: 80 2024-09-22T03:07:08.225 INFO:tasks.workunit.client.0.smithi039.stdout:read: 80 2024-09-22T03:07:08.225 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 80 2024-09-22T03:07:08.226 INFO:tasks.workunit.client.0.smithi039.stdout:opening testimg@orig 2024-09-22T03:07:08.226 INFO:tasks.workunit.client.0.smithi039.stdout:read: 80 2024-09-22T03:07:08.226 INFO:tasks.workunit.client.0.smithi039.stdout:write to snapshot returned -30 2024-09-22T03:07:08.226 INFO:tasks.workunit.client.0.smithi039.stdout:Read-only file system 2024-09-22T03:07:10.140 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 2 2024-09-22T03:07:10.140 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 2 2024-09-22T03:07:10.140 INFO:tasks.workunit.client.0.smithi039.stdout:snap: orig 2024-09-22T03:07:10.140 INFO:tasks.workunit.client.0.smithi039.stdout:snap: written 2024-09-22T03:07:10.140 INFO:tasks.workunit.client.0.smithi039.stdout:found orig with size 2097152 2024-09-22T03:07:10.140 INFO:tasks.workunit.client.0.smithi039.stdout:found written with size 2097152 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 1 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 1 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:snap: orig 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:found orig with size 2097152 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:num snaps is: 0 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:expected: 0 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (4020 ms) 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-09-22T03:07:10.141 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:12.157 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (2017 ms) 2024-09-22T03:07:12.157 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestClone 2024-09-22T03:07:12.157 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:6444: Skipped 2024-09-22T03:07:12.157 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:12.157 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestClone (0 ms) 2024-09-22T03:07:12.157 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestClone2 2024-09-22T03:07:12.157 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:6608: Skipped 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestClone2 (0 ms) 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ListChildren 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:6841: Skipped 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ListChildren (0 ms) 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:6995: Skipped 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered (0 ms) 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.LockingPP 2024-09-22T03:07:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:12.411 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.LockingPP (254 ms) 2024-09-22T03:07:12.411 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.FlushAio 2024-09-22T03:07:12.411 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:14.721 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.FlushAio (2310 ms) 2024-09-22T03:07:14.721 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-09-22T03:07:14.721 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:16.793 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.FlushAioPP (2072 ms) 2024-09-22T03:07:16.793 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-09-22T03:07:16.793 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:16.877 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (85 ms) 2024-09-22T03:07:16.878 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-09-22T03:07:16.878 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:16.983 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (105 ms) 2024-09-22T03:07:16.983 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-09-22T03:07:16.983 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:17.055 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.ZeroLengthRead (73 ms) 2024-09-22T03:07:17.056 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-09-22T03:07:17.056 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:17.207 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.LargeCacheRead (151 ms) 2024-09-22T03:07:17.207 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-09-22T03:07:17.207 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9152: Skipped 2024-09-22T03:07:17.207 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:17.207 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestPendingAio (0 ms) 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9264: Skipped 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyup (0 ms) 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9325: Skipped 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyupMismatch (0 ms) 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.Flatten 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9336: Skipped 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.Flatten (0 ms) 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.Sparsify 2024-09-22T03:07:17.208 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:17.322 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 32768 2024-09-22T03:07:17.322 INFO:tasks.workunit.client.0.smithi039.stdout:read: 32768 2024-09-22T03:07:17.322 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.Sparsify (115 ms) 2024-09-22T03:07:17.322 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-09-22T03:07:17.323 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:17.443 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.SparsifyPP (121 ms) 2024-09-22T03:07:17.443 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-09-22T03:07:17.443 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:21.961 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.SnapshotLimit (4518 ms) 2024-09-22T03:07:21.961 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-09-22T03:07:21.961 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:26.997 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5034 ms) 2024-09-22T03:07:26.997 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-09-22T03:07:26.997 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9523: Skipped 2024-09-22T03:07:26.997 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:26.997 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RebuildObjectMapViaLockOwner (0 ms) 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9575: Skipped 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RenameViaLockOwner (0 ms) 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9615: Skipped 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapCreateViaLockOwner (0 ms) 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-09-22T03:07:26.998 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9659: Skipped 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveViaLockOwner (0 ms) 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9699: Skipped 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.UpdateFeaturesViaLockOwner (0 ms) 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9736: Skipped 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-09-22T03:07:26.999 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9777: Skipped 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapRemove2 (0 ms) 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9823: Skipped 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapRenameViaLockOwner (0 ms) 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9864: Skipped 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapProtectViaLockOwner (0 ms) 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9905: Skipped 2024-09-22T03:07:27.000 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapUnprotectViaLockOwner (0 ms) 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9949: Skipped 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.FlattenViaLockOwner (0 ms) 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9997: Skipped 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ResizeViaLockOwner (0 ms) 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-09-22T03:07:27.001 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10033: Skipped 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SparsifyViaLockOwner (0 ms) 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10069: Skipped 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ObjectMapConsistentSnap (0 ms) 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.Metadata 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10121: Skipped 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.Metadata (0 ms) 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10310: Skipped 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.002 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.MetadataPP (0 ms) 2024-09-22T03:07:27.003 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-09-22T03:07:27.003 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10391: Skipped 2024-09-22T03:07:27.003 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.003 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.UpdateFeatures (0 ms) 2024-09-22T03:07:27.003 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-09-22T03:07:27.003 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2024-09-22T03:07:27.003 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-09-22T03:07:27.003 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:27.065 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.RebuildObjectMap (70 ms) 2024-09-22T03:07:27.065 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10584: Skipped 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RebuildNewObjectMap (0 ms) 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10609: Skipped 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CheckObjectMap (0 ms) 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-09-22T03:07:27.066 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:27.167 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.BlockingAIO (102 ms) 2024-09-22T03:07:27.167 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-09-22T03:07:27.167 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10737: Skipped 2024-09-22T03:07:27.167 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.167 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockTransition (0 ms) 2024-09-22T03:07:27.167 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-09-22T03:07:27.167 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10793: Skipped 2024-09-22T03:07:27.168 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.168 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-09-22T03:07:27.168 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-09-22T03:07:27.168 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10846: Skipped 2024-09-22T03:07:27.168 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:27.168 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CacheMayCopyOnWrite (0 ms) 2024-09-22T03:07:27.168 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-09-22T03:07:27.168 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:28.020 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (853 ms) 2024-09-22T03:07:28.020 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-09-22T03:07:32.095 INFO:tasks.workunit.client.0.smithi039.stdout:0.00% done 2024-09-22T03:07:32.095 INFO:tasks.workunit.client.0.smithi039.stdout:100.00% done 2024-09-22T03:07:32.095 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestImageOptions (4075 ms) 2024-09-22T03:07:32.095 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:0.00% done 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:100.00% done 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (1672 ms) 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-09-22T03:07:33.770 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:33.954 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:07:33.954 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:07:33.954 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:07:33.954 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:07:33.954 INFO:tasks.workunit.client.0.smithi039.stdout:wrote: 512 2024-09-22T03:07:33.954 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:07:33.955 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:started write 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:write completion cb called! 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 0 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:finished write 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:07:33.956 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:created completion 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:started read 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:read completion cb called! 2024-09-22T03:07:33.957 INFO:tasks.workunit.client.0.smithi039.stdout:return value is: 512 2024-09-22T03:07:33.958 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.ImagePollIO (187 ms) 2024-09-22T03:07:33.958 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.Mirror 2024-09-22T03:07:33.988 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:34.018 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:34.260 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.Mirror (306 ms) 2024-09-22T03:07:34.260 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (84 ms) 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11322: Skipped 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CreateWithMirrorEnabled (0 ms) 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11364: Skipped 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (0 ms) 2024-09-22T03:07:34.344 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11410: Skipped 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLock (0 ms) 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.BreakLock 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11553: Skipped 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.BreakLock (0 ms) 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-09-22T03:07:34.345 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (61 ms) 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11676: Skipped 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurge (0 ms) 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11722: Skipped 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (0 ms) 2024-09-22T03:07:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11753: Skipped 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestTrashPurge (0 ms) 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11810: Skipped 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndRestore (0 ms) 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-09-22T03:07:34.407 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:34.537 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestListWatchers (133 ms) 2024-09-22T03:07:34.538 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-09-22T03:07:34.538 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:35.453 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.TestSetSnapById (915 ms) 2024-09-22T03:07:35.454 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.Namespaces 2024-09-22T03:07:35.642 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.Namespaces (189 ms) 2024-09-22T03:07:35.642 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-09-22T03:07:35.872 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:35.880 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.NamespacesPP (238 ms) 2024-09-22T03:07:35.880 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.Migration 2024-09-22T03:07:35.880 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:35.880 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:38.357 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.Migration (2477 ms) 2024-09-22T03:07:38.357 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-09-22T03:07:38.357 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:38.357 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:40.845 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.MigrationPP (2487 ms) 2024-09-22T03:07:40.845 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-09-22T03:07:40.845 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12177: Skipped 2024-09-22T03:07:40.845 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.845 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestGetAccessTimestamp (0 ms) 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12202: Skipped 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestGetModifyTimestamp (0 ms) 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12225: Skipped 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ZeroOverlapFlatten (0 ms) 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12258: Skipped 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.PoolMetadata (0 ms) 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-09-22T03:07:40.846 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12370: Skipped 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.PoolMetadataPP (0 ms) 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.Config 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12422: Skipped 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.Config (0 ms) 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12506: Skipped 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ConfigPP (0 ms) 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12583: Skipped 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.847 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.PoolStatsPP (0 ms) 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12658: Skipped 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ImageSpec (0 ms) 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:12853: Skipped 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveWithChildMissing (0 ms) 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-09-22T03:07:40.848 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:47.069 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.QuiesceWatch (6225 ms) 2024-09-22T03:07:47.069 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-09-22T03:07:47.070 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:53.125 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (6056 ms) 2024-09-22T03:07:53.125 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-09-22T03:07:53.125 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:13115: Skipped 2024-09-22T03:07:53.125 INFO:tasks.workunit.client.0.smithi039.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:07:53.125 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError (0 ms) 2024-09-22T03:07:53.126 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-09-22T03:07:53.126 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:07:53.185 INFO:tasks.workunit.client.0.smithi039.stderr:test quiesce is not long enough to time out 2024-09-22T03:08:02.048 INFO:tasks.workunit.client.0.smithi039.stderr:test quiesce is timed out 2024-09-22T03:08:02.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 0 2024-09-22T03:08:03.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 1 2024-09-22T03:08:04.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 2 2024-09-22T03:08:05.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 3 2024-09-22T03:08:06.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 4 2024-09-22T03:08:07.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 5 2024-09-22T03:08:08.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 6 2024-09-22T03:08:09.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 7 2024-09-22T03:08:10.062 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 8 2024-09-22T03:08:11.063 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 9 2024-09-22T03:08:12.063 INFO:tasks.workunit.client.0.smithi039.stderr:waiting for timed out ... 10 2024-09-22T03:08:23.063 INFO:tasks.workunit.client.0.smithi039.stderr:test retry succeeds 2024-09-22T03:08:26.226 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (33101 ms) 2024-09-22T03:08:26.226 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-09-22T03:08:26.226 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:08:26.393 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.WriteZeroes (167 ms) 2024-09-22T03:08:26.393 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-09-22T03:08:26.393 INFO:tasks.workunit.client.0.smithi039.stdout:using old format 2024-09-22T03:08:26.583 INFO:tasks.workunit.client.0.smithi039.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (191 ms) 2024-09-22T03:08:26.584 INFO:tasks.workunit.client.0.smithi039.stdout:[ RUN ] TestLibRBD.ConcurrentOperations 2024-09-22T03:08:26.584 INFO:tasks.workunit.client.0.smithi039.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:13474: Skipped 2024-09-22T03:08:26.584 INFO:tasks.workunit.client.0.smithi039.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:08:26.584 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations (0 ms) 2024-09-22T03:08:32.991 INFO:tasks.workunit.client.0.smithi039.stdout:[----------] 170 tests from TestLibRBD (338605 ms total) 2024-09-22T03:08:32.991 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-22T03:08:32.991 INFO:tasks.workunit.client.0.smithi039.stdout:[----------] Global test environment tear-down 2024-09-22T03:08:32.991 INFO:tasks.workunit.client.0.smithi039.stdout:[==========] 170 tests from 1 test suite ran. (345048 ms total) 2024-09-22T03:08:32.991 INFO:tasks.workunit.client.0.smithi039.stdout:[ PASSED ] 103 tests. 2024-09-22T03:08:32.991 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] 66 tests, listed below: 2024-09-22T03:08:32.991 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool 2024-09-22T03:08:32.991 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespace 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData 2024-09-22T03:08:32.992 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestClone 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestClone2 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ListChildren 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestPendingAio 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyup 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.Flatten 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RenameViaLockOwner 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapCreateViaLockOwner 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveViaLockOwner 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-09-22T03:08:32.993 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapRemove2 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapRenameViaLockOwner 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapProtectViaLockOwner 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapUnprotectViaLockOwner 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.FlattenViaLockOwner 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ResizeViaLockOwner 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SparsifyViaLockOwner 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ObjectMapConsistentSnap 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.Metadata 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.MetadataPP 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.UpdateFeatures 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.RebuildNewObjectMap 2024-09-22T03:08:32.994 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CheckObjectMap 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockTransition 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CacheMayCopyOnWrite 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.CreateWithMirrorEnabled 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLock 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.BreakLock 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurge 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestTrashPurge 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestTrashMoveAndRestore 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestGetAccessTimestamp 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.TestGetModifyTimestamp 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ZeroOverlapFlatten 2024-09-22T03:08:32.995 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.PoolMetadata 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.PoolMetadataPP 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.Config 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ConfigPP 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.PoolStatsPP 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ImageSpec 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.SnapRemoveWithChildMissing 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ FAILED ] 1 test, listed below: 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout:[ FAILED ] TestLibRBD.TestEncryptionLUKS2 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout: 1 FAILED TEST 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout: YOU HAVE 1 DISABLED TEST 2024-09-22T03:08:32.996 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-22T03:08:32.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:08:32.999 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2024-09-22T03:08:32.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-22T03:08:33.357 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-09-22T03:08:40.407 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bc51ec3968594bee92c25ac23d1331cf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-09-22T03:08:40.410 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T03:08:40.418 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:08:40.418 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:08:40.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:40.540+0000 7f9d30917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:40.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:40.547+0000 7f9d30917640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:40.733 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:08:40.734 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-22T03:08:40.747 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-22T03:08:39.719630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5220,"kb_used_data":5220,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156887964,"statfs":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168071421,"num_objects":-220,"num_object_clones":0,"num_object_copies":-440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28121,"num_read_kb":-145507,"num_write":-1507,"num_write_kb":-202236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001547"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.268428+0000","last_change":"2024-09-22T03:01:45.758866+0000","last_active":"2024-09-22T03:08:33.268428+0000","last_peered":"2024-09-22T03:08:33.268428+0000","last_clean":"2024-09-22T03:08:33.268428+0000","last_became_active":"2024-09-22T03:01:43.765548+0000","last_became_peered":"2024-09-22T03:01:43.765548+0000","last_unstale":"2024-09-22T03:08:33.268428+0000","last_undegraded":"2024-09-22T03:08:33.268428+0000","last_fullsized":"2024-09-22T03:08:33.268428+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.029899+0000","last_change":"2024-09-22T03:01:45.766286+0000","last_active":"2024-09-22T03:08:33.029899+0000","last_peered":"2024-09-22T03:08:33.029899+0000","last_clean":"2024-09-22T03:08:33.029899+0000","last_became_active":"2024-09-22T03:01:43.786726+0000","last_became_peered":"2024-09-22T03:01:43.786726+0000","last_unstale":"2024-09-22T03:08:33.029899+0000","last_undegraded":"2024-09-22T03:08:33.029899+0000","last_fullsized":"2024-09-22T03:08:33.029899+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.055835+0000","last_change":"2024-09-22T03:01:45.767686+0000","last_active":"2024-09-22T03:08:33.055835+0000","last_peered":"2024-09-22T03:08:33.055835+0000","last_clean":"2024-09-22T03:08:33.055835+0000","last_became_active":"2024-09-22T03:01:43.774106+0000","last_became_peered":"2024-09-22T03:01:43.774106+0000","last_unstale":"2024-09-22T03:08:33.055835+0000","last_undegraded":"2024-09-22T03:08:33.055835+0000","last_fullsized":"2024-09-22T03:08:33.055835+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.036223+0000","last_change":"2024-09-22T03:01:45.767534+0000","last_active":"2024-09-22T03:08:33.036223+0000","last_peered":"2024-09-22T03:08:33.036223+0000","last_clean":"2024-09-22T03:08:33.036223+0000","last_became_active":"2024-09-22T03:01:43.777551+0000","last_became_peered":"2024-09-22T03:01:43.777551+0000","last_unstale":"2024-09-22T03:08:33.036223+0000","last_undegraded":"2024-09-22T03:08:33.036223+0000","last_fullsized":"2024-09-22T03:08:33.036223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.489118+0000","last_change":"2024-09-22T03:01:45.760741+0000","last_active":"2024-09-22T03:08:33.489118+0000","last_peered":"2024-09-22T03:08:33.489118+0000","last_clean":"2024-09-22T03:08:33.489118+0000","last_became_active":"2024-09-22T03:01:43.844167+0000","last_became_peered":"2024-09-22T03:01:43.844167+0000","last_unstale":"2024-09-22T03:08:33.489118+0000","last_undegraded":"2024-09-22T03:08:33.489118+0000","last_fullsized":"2024-09-22T03:08:33.489118+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.066705+0000","last_change":"2024-09-22T03:01:45.762398+0000","last_active":"2024-09-22T03:08:33.066705+0000","last_peered":"2024-09-22T03:08:33.066705+0000","last_clean":"2024-09-22T03:08:33.066705+0000","last_became_active":"2024-09-22T03:01:43.814273+0000","last_became_peered":"2024-09-22T03:01:43.814273+0000","last_unstale":"2024-09-22T03:08:33.066705+0000","last_undegraded":"2024-09-22T03:08:33.066705+0000","last_fullsized":"2024-09-22T03:08:33.066705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.073757+0000","last_change":"2024-09-22T03:01:45.758763+0000","last_active":"2024-09-22T03:08:33.073757+0000","last_peered":"2024-09-22T03:08:33.073757+0000","last_clean":"2024-09-22T03:08:33.073757+0000","last_became_active":"2024-09-22T03:01:43.789338+0000","last_became_peered":"2024-09-22T03:01:43.789338+0000","last_unstale":"2024-09-22T03:08:33.073757+0000","last_undegraded":"2024-09-22T03:08:33.073757+0000","last_fullsized":"2024-09-22T03:08:33.073757+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":334,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.054339+0000","last_change":"2024-09-22T03:01:45.767160+0000","last_active":"2024-09-22T03:08:33.054339+0000","last_peered":"2024-09-22T03:08:33.054339+0000","last_clean":"2024-09-22T03:08:33.054339+0000","last_became_active":"2024-09-22T03:01:43.756369+0000","last_became_peered":"2024-09-22T03:01:43.756369+0000","last_unstale":"2024-09-22T03:08:33.054339+0000","last_undegraded":"2024-09-22T03:08:33.054339+0000","last_fullsized":"2024-09-22T03:08:33.054339+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":462,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.046498+0000","last_change":"2024-09-22T03:01:42.751062+0000","last_active":"2024-09-22T03:08:33.046498+0000","last_peered":"2024-09-22T03:08:33.046498+0000","last_clean":"2024-09-22T03:08:33.046498+0000","last_became_active":"2024-09-22T03:01:42.736481+0000","last_became_peered":"2024-09-22T03:01:42.736481+0000","last_unstale":"2024-09-22T03:08:33.046498+0000","last_undegraded":"2024-09-22T03:08:33.046498+0000","last_fullsized":"2024-09-22T03:08:33.046498+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_clean_scrub_stamp":"2024-09-22T03:01:41.680033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1052,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296676,"statfs":{"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2228,"kb_used_data":2228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295500,"statfs":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1940,"kb_used_data":1940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295788,"statfs":{"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:08:40.748 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:08:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:40.849+0000 7fc562d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:40.856+0000 7fc562d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:41.042 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:08:41.042 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-22T03:08:41.054 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-22T03:08:39.719630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5220,"kb_used_data":5220,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156887964,"statfs":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168071421,"num_objects":-220,"num_object_clones":0,"num_object_copies":-440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28121,"num_read_kb":-145507,"num_write":-1507,"num_write_kb":-202236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001547"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.268428+0000","last_change":"2024-09-22T03:01:45.758866+0000","last_active":"2024-09-22T03:08:33.268428+0000","last_peered":"2024-09-22T03:08:33.268428+0000","last_clean":"2024-09-22T03:08:33.268428+0000","last_became_active":"2024-09-22T03:01:43.765548+0000","last_became_peered":"2024-09-22T03:01:43.765548+0000","last_unstale":"2024-09-22T03:08:33.268428+0000","last_undegraded":"2024-09-22T03:08:33.268428+0000","last_fullsized":"2024-09-22T03:08:33.268428+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.029899+0000","last_change":"2024-09-22T03:01:45.766286+0000","last_active":"2024-09-22T03:08:33.029899+0000","last_peered":"2024-09-22T03:08:33.029899+0000","last_clean":"2024-09-22T03:08:33.029899+0000","last_became_active":"2024-09-22T03:01:43.786726+0000","last_became_peered":"2024-09-22T03:01:43.786726+0000","last_unstale":"2024-09-22T03:08:33.029899+0000","last_undegraded":"2024-09-22T03:08:33.029899+0000","last_fullsized":"2024-09-22T03:08:33.029899+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.055835+0000","last_change":"2024-09-22T03:01:45.767686+0000","last_active":"2024-09-22T03:08:33.055835+0000","last_peered":"2024-09-22T03:08:33.055835+0000","last_clean":"2024-09-22T03:08:33.055835+0000","last_became_active":"2024-09-22T03:01:43.774106+0000","last_became_peered":"2024-09-22T03:01:43.774106+0000","last_unstale":"2024-09-22T03:08:33.055835+0000","last_undegraded":"2024-09-22T03:08:33.055835+0000","last_fullsized":"2024-09-22T03:08:33.055835+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.036223+0000","last_change":"2024-09-22T03:01:45.767534+0000","last_active":"2024-09-22T03:08:33.036223+0000","last_peered":"2024-09-22T03:08:33.036223+0000","last_clean":"2024-09-22T03:08:33.036223+0000","last_became_active":"2024-09-22T03:01:43.777551+0000","last_became_peered":"2024-09-22T03:01:43.777551+0000","last_unstale":"2024-09-22T03:08:33.036223+0000","last_undegraded":"2024-09-22T03:08:33.036223+0000","last_fullsized":"2024-09-22T03:08:33.036223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.489118+0000","last_change":"2024-09-22T03:01:45.760741+0000","last_active":"2024-09-22T03:08:33.489118+0000","last_peered":"2024-09-22T03:08:33.489118+0000","last_clean":"2024-09-22T03:08:33.489118+0000","last_became_active":"2024-09-22T03:01:43.844167+0000","last_became_peered":"2024-09-22T03:01:43.844167+0000","last_unstale":"2024-09-22T03:08:33.489118+0000","last_undegraded":"2024-09-22T03:08:33.489118+0000","last_fullsized":"2024-09-22T03:08:33.489118+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.066705+0000","last_change":"2024-09-22T03:01:45.762398+0000","last_active":"2024-09-22T03:08:33.066705+0000","last_peered":"2024-09-22T03:08:33.066705+0000","last_clean":"2024-09-22T03:08:33.066705+0000","last_became_active":"2024-09-22T03:01:43.814273+0000","last_became_peered":"2024-09-22T03:01:43.814273+0000","last_unstale":"2024-09-22T03:08:33.066705+0000","last_undegraded":"2024-09-22T03:08:33.066705+0000","last_fullsized":"2024-09-22T03:08:33.066705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.073757+0000","last_change":"2024-09-22T03:01:45.758763+0000","last_active":"2024-09-22T03:08:33.073757+0000","last_peered":"2024-09-22T03:08:33.073757+0000","last_clean":"2024-09-22T03:08:33.073757+0000","last_became_active":"2024-09-22T03:01:43.789338+0000","last_became_peered":"2024-09-22T03:01:43.789338+0000","last_unstale":"2024-09-22T03:08:33.073757+0000","last_undegraded":"2024-09-22T03:08:33.073757+0000","last_fullsized":"2024-09-22T03:08:33.073757+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":334,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.054339+0000","last_change":"2024-09-22T03:01:45.767160+0000","last_active":"2024-09-22T03:08:33.054339+0000","last_peered":"2024-09-22T03:08:33.054339+0000","last_clean":"2024-09-22T03:08:33.054339+0000","last_became_active":"2024-09-22T03:01:43.756369+0000","last_became_peered":"2024-09-22T03:01:43.756369+0000","last_unstale":"2024-09-22T03:08:33.054339+0000","last_undegraded":"2024-09-22T03:08:33.054339+0000","last_fullsized":"2024-09-22T03:08:33.054339+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":462,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.046498+0000","last_change":"2024-09-22T03:01:42.751062+0000","last_active":"2024-09-22T03:08:33.046498+0000","last_peered":"2024-09-22T03:08:33.046498+0000","last_clean":"2024-09-22T03:08:33.046498+0000","last_became_active":"2024-09-22T03:01:42.736481+0000","last_became_peered":"2024-09-22T03:01:42.736481+0000","last_unstale":"2024-09-22T03:08:33.046498+0000","last_undegraded":"2024-09-22T03:08:33.046498+0000","last_fullsized":"2024-09-22T03:08:33.046498+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_clean_scrub_stamp":"2024-09-22T03:01:41.680033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1052,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296676,"statfs":{"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2228,"kb_used_data":2228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295500,"statfs":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1940,"kb_used_data":1940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295788,"statfs":{"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:08:41.055 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:08:41.055 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:08:41.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:41.157+0000 7f8715a58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:41.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:41.163+0000 7f8715a58640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:41.351 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:08:41.352 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-22T03:08:41.364 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-22T03:08:39.719630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5220,"kb_used_data":5220,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156887964,"statfs":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168071421,"num_objects":-220,"num_object_clones":0,"num_object_copies":-440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-28121,"num_read_kb":-145507,"num_write":-1507,"num_write_kb":-202236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-75,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001547"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.268428+0000","last_change":"2024-09-22T03:01:45.758866+0000","last_active":"2024-09-22T03:08:33.268428+0000","last_peered":"2024-09-22T03:08:33.268428+0000","last_clean":"2024-09-22T03:08:33.268428+0000","last_became_active":"2024-09-22T03:01:43.765548+0000","last_became_peered":"2024-09-22T03:01:43.765548+0000","last_unstale":"2024-09-22T03:08:33.268428+0000","last_undegraded":"2024-09-22T03:08:33.268428+0000","last_fullsized":"2024-09-22T03:08:33.268428+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.029899+0000","last_change":"2024-09-22T03:01:45.766286+0000","last_active":"2024-09-22T03:08:33.029899+0000","last_peered":"2024-09-22T03:08:33.029899+0000","last_clean":"2024-09-22T03:08:33.029899+0000","last_became_active":"2024-09-22T03:01:43.786726+0000","last_became_peered":"2024-09-22T03:01:43.786726+0000","last_unstale":"2024-09-22T03:08:33.029899+0000","last_undegraded":"2024-09-22T03:08:33.029899+0000","last_fullsized":"2024-09-22T03:08:33.029899+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.055835+0000","last_change":"2024-09-22T03:01:45.767686+0000","last_active":"2024-09-22T03:08:33.055835+0000","last_peered":"2024-09-22T03:08:33.055835+0000","last_clean":"2024-09-22T03:08:33.055835+0000","last_became_active":"2024-09-22T03:01:43.774106+0000","last_became_peered":"2024-09-22T03:01:43.774106+0000","last_unstale":"2024-09-22T03:08:33.055835+0000","last_undegraded":"2024-09-22T03:08:33.055835+0000","last_fullsized":"2024-09-22T03:08:33.055835+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.036223+0000","last_change":"2024-09-22T03:01:45.767534+0000","last_active":"2024-09-22T03:08:33.036223+0000","last_peered":"2024-09-22T03:08:33.036223+0000","last_clean":"2024-09-22T03:08:33.036223+0000","last_became_active":"2024-09-22T03:01:43.777551+0000","last_became_peered":"2024-09-22T03:01:43.777551+0000","last_unstale":"2024-09-22T03:08:33.036223+0000","last_undegraded":"2024-09-22T03:08:33.036223+0000","last_fullsized":"2024-09-22T03:08:33.036223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.489118+0000","last_change":"2024-09-22T03:01:45.760741+0000","last_active":"2024-09-22T03:08:33.489118+0000","last_peered":"2024-09-22T03:08:33.489118+0000","last_clean":"2024-09-22T03:08:33.489118+0000","last_became_active":"2024-09-22T03:01:43.844167+0000","last_became_peered":"2024-09-22T03:01:43.844167+0000","last_unstale":"2024-09-22T03:08:33.489118+0000","last_undegraded":"2024-09-22T03:08:33.489118+0000","last_fullsized":"2024-09-22T03:08:33.489118+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.066705+0000","last_change":"2024-09-22T03:01:45.762398+0000","last_active":"2024-09-22T03:08:33.066705+0000","last_peered":"2024-09-22T03:08:33.066705+0000","last_clean":"2024-09-22T03:08:33.066705+0000","last_became_active":"2024-09-22T03:01:43.814273+0000","last_became_peered":"2024-09-22T03:01:43.814273+0000","last_unstale":"2024-09-22T03:08:33.066705+0000","last_undegraded":"2024-09-22T03:08:33.066705+0000","last_fullsized":"2024-09-22T03:08:33.066705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.073757+0000","last_change":"2024-09-22T03:01:45.758763+0000","last_active":"2024-09-22T03:08:33.073757+0000","last_peered":"2024-09-22T03:08:33.073757+0000","last_clean":"2024-09-22T03:08:33.073757+0000","last_became_active":"2024-09-22T03:01:43.789338+0000","last_became_peered":"2024-09-22T03:01:43.789338+0000","last_unstale":"2024-09-22T03:08:33.073757+0000","last_undegraded":"2024-09-22T03:08:33.073757+0000","last_fullsized":"2024-09-22T03:08:33.073757+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":334,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.054339+0000","last_change":"2024-09-22T03:01:45.767160+0000","last_active":"2024-09-22T03:08:33.054339+0000","last_peered":"2024-09-22T03:08:33.054339+0000","last_clean":"2024-09-22T03:08:33.054339+0000","last_became_active":"2024-09-22T03:01:43.756369+0000","last_became_peered":"2024-09-22T03:01:43.756369+0000","last_unstale":"2024-09-22T03:08:33.054339+0000","last_undegraded":"2024-09-22T03:08:33.054339+0000","last_fullsized":"2024-09-22T03:08:33.054339+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":462,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.046498+0000","last_change":"2024-09-22T03:01:42.751062+0000","last_active":"2024-09-22T03:08:33.046498+0000","last_peered":"2024-09-22T03:08:33.046498+0000","last_clean":"2024-09-22T03:08:33.046498+0000","last_became_active":"2024-09-22T03:01:42.736481+0000","last_became_peered":"2024-09-22T03:01:42.736481+0000","last_unstale":"2024-09-22T03:08:33.046498+0000","last_undegraded":"2024-09-22T03:08:33.046498+0000","last_fullsized":"2024-09-22T03:08:33.046498+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_clean_scrub_stamp":"2024-09-22T03:01:41.680033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1052,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296676,"statfs":{"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2228,"kb_used_data":2228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295500,"statfs":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738783,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1940,"kb_used_data":1940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295788,"statfs":{"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:08:41.365 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:08:41.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:41.467+0000 7f2116997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:41.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:41.473+0000 7f2116997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:41.664 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:08:41.664 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":119,"fsid":"df5703b9-69b9-43c4-b0b0-b568274428f6","created":"2024-09-22T03:01:32.774495+0000","modified":"2024-09-22T03:08:32.981055+0000","last_up_change":"2024-09-22T03:01:39.668223+0000","last_in_change":"2024-09-22T03:01:34.674342+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:01:41.669564+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:01:42.607262+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1471c1f6-ee1d-4ea6-ae1f-1dd37999c2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":2356902669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":2356902669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":2356902669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":2356902669}]},"public_addr":"172.21.15.39:6806/2356902669","cluster_addr":"172.21.15.39:6803/2356902669","heartbeat_back_addr":"172.21.15.39:6812/2356902669","heartbeat_front_addr":"172.21.15.39:6811/2356902669","state":["exists","up"]},{"osd":1,"uuid":"f6b2e13d-3702-4ba6-b46b-cd4963228832","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":3532051251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3532051251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3532051251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3532051251}]},"public_addr":"172.21.15.39:6805/3532051251","cluster_addr":"172.21.15.39:6804/3532051251","heartbeat_back_addr":"172.21.15.39:6809/3532051251","heartbeat_front_addr":"172.21.15.39:6808/3532051251","state":["exists","up"]},{"osd":2,"uuid":"2f051db8-ce8b-4dd3-99b9-abe7cacabea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":1848712837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":1848712837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1848712837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":1848712837}]},"public_addr":"172.21.15.39:6802/1848712837","cluster_addr":"172.21.15.39:6807/1848712837","heartbeat_back_addr":"172.21.15.39:6810/1848712837","heartbeat_front_addr":"172.21.15.39:6813/1848712837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":13},{"begin":17,"length":1},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":28,"length":8}]},{"pool":4,"snaps":[{"begin":2,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":8,"snaps":[{"begin":2,"length":4}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:08:42.678 INFO:tasks.ceph:Scrubbing osd.0 2024-09-22T03:08:42.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:08:42.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:42.795+0000 7f968730e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:42.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:42.805+0000 7f968730e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:42.819 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-22T03:08:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "" 2024-09-22T03:08:42.819 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-22T03:08:42.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-22T03:08:42.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:42.933+0000 7f8900997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:42.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:42.940+0000 7f8900997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:43.128 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 0 to deep-scrub 2024-09-22T03:08:43.145 INFO:tasks.ceph:Scrubbing osd.1 2024-09-22T03:08:43.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:08:43.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:43.246+0000 7f9221c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:43.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:43.257+0000 7f9221c6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:43.270 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-22T03:08:43.270 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "" 2024-09-22T03:08:43.270 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-22T03:08:43.280 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-22T03:08:43.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:43.382+0000 7f051f957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:43.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:43.389+0000 7f051f957640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:43.575 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 1 to deep-scrub 2024-09-22T03:08:43.592 INFO:tasks.ceph:Scrubbing osd.2 2024-09-22T03:08:43.593 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:08:43.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:43.696+0000 7f54c5e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:43.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:43.706+0000 7f54c5e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:43.719 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-22T03:08:43.719 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "" 2024-09-22T03:08:43.719 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-22T03:08:43.730 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-22T03:08:43.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:43.831+0000 7f31c5357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:43.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:43.839+0000 7f31c5357640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:44.030 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 2 to deep-scrub 2024-09-22T03:08:44.048 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:08:44.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:44.150+0000 7f4046e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:44.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:08:44.156+0000 7f4046e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:08:44.340 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:08:44.340 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-22T03:08:44.353 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-22T03:08:43.720632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5220,"kb_used_data":5220,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156887964,"statfs":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.736750"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.268428+0000","last_change":"2024-09-22T03:01:45.758866+0000","last_active":"2024-09-22T03:08:33.268428+0000","last_peered":"2024-09-22T03:08:33.268428+0000","last_clean":"2024-09-22T03:08:33.268428+0000","last_became_active":"2024-09-22T03:01:43.765548+0000","last_became_peered":"2024-09-22T03:01:43.765548+0000","last_unstale":"2024-09-22T03:08:33.268428+0000","last_undegraded":"2024-09-22T03:08:33.268428+0000","last_fullsized":"2024-09-22T03:08:33.268428+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.029899+0000","last_change":"2024-09-22T03:01:45.766286+0000","last_active":"2024-09-22T03:08:33.029899+0000","last_peered":"2024-09-22T03:08:33.029899+0000","last_clean":"2024-09-22T03:08:33.029899+0000","last_became_active":"2024-09-22T03:01:43.786726+0000","last_became_peered":"2024-09-22T03:01:43.786726+0000","last_unstale":"2024-09-22T03:08:33.029899+0000","last_undegraded":"2024-09-22T03:08:33.029899+0000","last_fullsized":"2024-09-22T03:08:33.029899+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.055835+0000","last_change":"2024-09-22T03:01:45.767686+0000","last_active":"2024-09-22T03:08:33.055835+0000","last_peered":"2024-09-22T03:08:33.055835+0000","last_clean":"2024-09-22T03:08:33.055835+0000","last_became_active":"2024-09-22T03:01:43.774106+0000","last_became_peered":"2024-09-22T03:01:43.774106+0000","last_unstale":"2024-09-22T03:08:33.055835+0000","last_undegraded":"2024-09-22T03:08:33.055835+0000","last_fullsized":"2024-09-22T03:08:33.055835+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.036223+0000","last_change":"2024-09-22T03:01:45.767534+0000","last_active":"2024-09-22T03:08:33.036223+0000","last_peered":"2024-09-22T03:08:33.036223+0000","last_clean":"2024-09-22T03:08:33.036223+0000","last_became_active":"2024-09-22T03:01:43.777551+0000","last_became_peered":"2024-09-22T03:01:43.777551+0000","last_unstale":"2024-09-22T03:08:33.036223+0000","last_undegraded":"2024-09-22T03:08:33.036223+0000","last_fullsized":"2024-09-22T03:08:33.036223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.489118+0000","last_change":"2024-09-22T03:01:45.760741+0000","last_active":"2024-09-22T03:08:33.489118+0000","last_peered":"2024-09-22T03:08:33.489118+0000","last_clean":"2024-09-22T03:08:33.489118+0000","last_became_active":"2024-09-22T03:01:43.844167+0000","last_became_peered":"2024-09-22T03:01:43.844167+0000","last_unstale":"2024-09-22T03:08:33.489118+0000","last_undegraded":"2024-09-22T03:08:33.489118+0000","last_fullsized":"2024-09-22T03:08:33.489118+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.066705+0000","last_change":"2024-09-22T03:01:45.762398+0000","last_active":"2024-09-22T03:08:33.066705+0000","last_peered":"2024-09-22T03:08:33.066705+0000","last_clean":"2024-09-22T03:08:33.066705+0000","last_became_active":"2024-09-22T03:01:43.814273+0000","last_became_peered":"2024-09-22T03:01:43.814273+0000","last_unstale":"2024-09-22T03:08:33.066705+0000","last_undegraded":"2024-09-22T03:08:33.066705+0000","last_fullsized":"2024-09-22T03:08:33.066705+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":333,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.073757+0000","last_change":"2024-09-22T03:01:45.758763+0000","last_active":"2024-09-22T03:08:33.073757+0000","last_peered":"2024-09-22T03:08:33.073757+0000","last_clean":"2024-09-22T03:08:33.073757+0000","last_became_active":"2024-09-22T03:01:43.789338+0000","last_became_peered":"2024-09-22T03:01:43.789338+0000","last_unstale":"2024-09-22T03:08:33.073757+0000","last_undegraded":"2024-09-22T03:08:33.073757+0000","last_fullsized":"2024-09-22T03:08:33.073757+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":334,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.054339+0000","last_change":"2024-09-22T03:01:45.767160+0000","last_active":"2024-09-22T03:08:33.054339+0000","last_peered":"2024-09-22T03:08:33.054339+0000","last_clean":"2024-09-22T03:08:33.054339+0000","last_became_active":"2024-09-22T03:01:43.756369+0000","last_became_peered":"2024-09-22T03:01:43.756369+0000","last_unstale":"2024-09-22T03:08:33.054339+0000","last_undegraded":"2024-09-22T03:08:33.054339+0000","last_fullsized":"2024-09-22T03:08:33.054339+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:42.689620+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":462,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:33.046498+0000","last_change":"2024-09-22T03:01:42.751062+0000","last_active":"2024-09-22T03:08:33.046498+0000","last_peered":"2024-09-22T03:08:33.046498+0000","last_clean":"2024-09-22T03:08:33.046498+0000","last_became_active":"2024-09-22T03:01:42.736481+0000","last_became_peered":"2024-09-22T03:01:42.736481+0000","last_unstale":"2024-09-22T03:08:33.046498+0000","last_undegraded":"2024-09-22T03:08:33.046498+0000","last_fullsized":"2024-09-22T03:08:33.046498+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:01:41.680033+0000","last_clean_scrub_stamp":"2024-09-22T03:01:41.680033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1052,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296676,"statfs":{"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2228,"kb_used_data":2228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295500,"statfs":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738788,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1940,"kb_used_data":1940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295788,"statfs":{"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:08:44.356 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-22T03:01:42.689620+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=42, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.356 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-22T03:01:42.689620+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=42, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.356 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-22T03:01:42.689620+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=42, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.356 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-22T03:01:42.689620+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=42, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.356 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-22T03:01:42.689620+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=42, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.357 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-22T03:01:42.689620+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=42, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.357 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-22T03:01:42.689620+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=42, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.357 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-22T03:01:42.689620+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=42, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.357 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-22T03:01:41.680033+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=1, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=8, tm_sec=41, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:08:44.357 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-22T03:09:04.358 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:09:04.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:09:04.474+0000 7fc6d58d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:04.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:09:04.481+0000 7fc6d58d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:04.665 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-22T03:09:04.666 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-22T03:09:04.678 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-22T03:09:03.725105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":5220,"kb_used_data":5220,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156887964,"statfs":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002825"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:43.577622+0000","last_change":"2024-09-22T03:08:43.589024+0000","last_active":"2024-09-22T03:08:43.577622+0000","last_peered":"2024-09-22T03:08:43.577622+0000","last_clean":"2024-09-22T03:08:43.577622+0000","last_became_active":"2024-09-22T03:01:43.765548+0000","last_became_peered":"2024-09-22T03:01:43.765548+0000","last_unstale":"2024-09-22T03:08:43.577622+0000","last_undegraded":"2024-09-22T03:08:43.577622+0000","last_fullsized":"2024-09-22T03:08:43.577622+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:43.588924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:43.588924+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:43.579011+0000","last_change":"2024-09-22T03:08:43.599832+0000","last_active":"2024-09-22T03:08:43.579011+0000","last_peered":"2024-09-22T03:08:43.579011+0000","last_clean":"2024-09-22T03:08:43.579011+0000","last_became_active":"2024-09-22T03:01:43.786726+0000","last_became_peered":"2024-09-22T03:01:43.786726+0000","last_unstale":"2024-09-22T03:08:43.579011+0000","last_undegraded":"2024-09-22T03:08:43.579011+0000","last_fullsized":"2024-09-22T03:08:43.579011+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:43.599750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:43.599750+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:43.578415+0000","last_change":"2024-09-22T03:08:43.596990+0000","last_active":"2024-09-22T03:08:43.578415+0000","last_peered":"2024-09-22T03:08:43.578415+0000","last_clean":"2024-09-22T03:08:43.578415+0000","last_became_active":"2024-09-22T03:01:43.774106+0000","last_became_peered":"2024-09-22T03:01:43.774106+0000","last_unstale":"2024-09-22T03:08:43.578415+0000","last_undegraded":"2024-09-22T03:08:43.578415+0000","last_fullsized":"2024-09-22T03:08:43.578415+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:43.596845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:43.596845+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:43.577812+0000","last_change":"2024-09-22T03:08:43.599271+0000","last_active":"2024-09-22T03:08:43.577812+0000","last_peered":"2024-09-22T03:08:43.577812+0000","last_clean":"2024-09-22T03:08:43.577812+0000","last_became_active":"2024-09-22T03:01:43.777551+0000","last_became_peered":"2024-09-22T03:01:43.777551+0000","last_unstale":"2024-09-22T03:08:43.577812+0000","last_undegraded":"2024-09-22T03:08:43.577812+0000","last_fullsized":"2024-09-22T03:08:43.577812+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:43.599118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:43.599118+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":337,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:43.128295+0000","last_change":"2024-09-22T03:08:43.143307+0000","last_active":"2024-09-22T03:08:43.128295+0000","last_peered":"2024-09-22T03:08:43.128295+0000","last_clean":"2024-09-22T03:08:43.128295+0000","last_became_active":"2024-09-22T03:01:43.844167+0000","last_became_peered":"2024-09-22T03:01:43.844167+0000","last_unstale":"2024-09-22T03:08:43.128295+0000","last_undegraded":"2024-09-22T03:08:43.128295+0000","last_fullsized":"2024-09-22T03:08:43.128295+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'2","last_scrub_stamp":"2024-09-22T03:08:43.143199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:43.143199+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:44.032530+0000","last_change":"2024-09-22T03:08:44.043911+0000","last_active":"2024-09-22T03:08:44.032530+0000","last_peered":"2024-09-22T03:08:44.032530+0000","last_clean":"2024-09-22T03:08:44.032530+0000","last_became_active":"2024-09-22T03:01:43.814273+0000","last_became_peered":"2024-09-22T03:01:43.814273+0000","last_unstale":"2024-09-22T03:08:44.032530+0000","last_undegraded":"2024-09-22T03:08:44.032530+0000","last_fullsized":"2024-09-22T03:08:44.032530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:44.043805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:44.043805+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":335,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:44.031131+0000","last_change":"2024-09-22T03:08:44.042800+0000","last_active":"2024-09-22T03:08:44.031131+0000","last_peered":"2024-09-22T03:08:44.031131+0000","last_clean":"2024-09-22T03:08:44.031131+0000","last_became_active":"2024-09-22T03:01:43.789338+0000","last_became_peered":"2024-09-22T03:01:43.789338+0000","last_unstale":"2024-09-22T03:08:44.031131+0000","last_undegraded":"2024-09-22T03:08:44.031131+0000","last_fullsized":"2024-09-22T03:08:44.031131+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:08:44.042703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:44.042703+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":336,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:43.577278+0000","last_change":"2024-09-22T03:08:43.595952+0000","last_active":"2024-09-22T03:08:43.577278+0000","last_peered":"2024-09-22T03:08:43.577278+0000","last_clean":"2024-09-22T03:08:43.577278+0000","last_became_active":"2024-09-22T03:01:43.756369+0000","last_became_peered":"2024-09-22T03:01:43.756369+0000","last_unstale":"2024-09-22T03:08:43.577278+0000","last_undegraded":"2024-09-22T03:08:43.577278+0000","last_fullsized":"2024-09-22T03:08:43.577278+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'1","last_scrub_stamp":"2024-09-22T03:08:43.595755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:43.595755+0000","last_clean_scrub_stamp":"2024-09-22T03:01:42.689620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":464,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-22T03:08:43.576137+0000","last_change":"2024-09-22T03:08:43.605649+0000","last_active":"2024-09-22T03:08:43.576137+0000","last_peered":"2024-09-22T03:08:43.576137+0000","last_clean":"2024-09-22T03:08:43.576137+0000","last_became_active":"2024-09-22T03:01:42.736481+0000","last_became_peered":"2024-09-22T03:01:42.736481+0000","last_unstale":"2024-09-22T03:08:43.576137+0000","last_undegraded":"2024-09-22T03:08:43.576137+0000","last_fullsized":"2024-09-22T03:08:43.576137+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'97","last_scrub_stamp":"2024-09-22T03:08:43.605515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:08:43.605515+0000","last_clean_scrub_stamp":"2024-09-22T03:01:41.680033+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653275136,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1052,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296676,"statfs":{"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738808,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2228,"kb_used_data":2228,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295500,"statfs":{"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738808,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1940,"kb_used_data":1940,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295788,"statfs":{"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550886912,"internally_reserved":0,"allocated":1986560,"data_stored":1986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550592000,"internally_reserved":0,"allocated":2281472,"data_stored":2281472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551796224,"internally_reserved":0,"allocated":1077248,"data_stored":1077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:09:04.679 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-22T03:09:04.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:09:04.777+0000 7fabef04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:04.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-22T03:09:04.787+0000 7fabef04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:09:04.993 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T03:09:04.993 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T03:09:04.993 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:09:04.993 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:09:07.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T03:09:11.095 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:09:11.095 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:09:11.095 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:09:13.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:09:13.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T03:09:17.198 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:09:17.198 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:09:17.198 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:09:18.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:09:18.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:09:23.299 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:09:23.300 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T03:09:23.300 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-22T03:09:23.300 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:09:23.335 INFO:tasks.ceph.mgr.x:Stopped 2024-09-22T03:09:23.336 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T03:09:23.336 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-22T03:09:23.336 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:09:23.391 INFO:tasks.ceph.mon.a:Stopped 2024-09-22T03:09:23.391 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T03:09:23.392 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(POOL_FULL\)' | head -n 1 2024-09-22T03:09:23.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:09:23.468 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T03:09:23.468 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-22T03:09:23.593 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T03:09:23.593 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-22T03:09:23.695 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T03:09:23.696 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-22T03:09:23.751 INFO:tasks.ceph:Archiving mon data... 2024-09-22T03:09:23.752 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914717/data/mon.a.tgz 2024-09-22T03:09:23.752 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-22T03:09:23.771 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.aLUCLUItDo 2024-09-22T03:09:23.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.aLUCLUItDo 2024-09-22T03:09:23.911 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.aLUCLUItDo 2024-09-22T03:09:23.996 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.aLUCLUItDo is 306KB 2024-09-22T03:09:24.090 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.aLUCLUItDo 2024-09-22T03:09:24.105 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T03:09:24.105 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:09:24.226 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:09:24.227 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T03:09:24.228 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914717/remote/smithi039/crash 2024-09-22T03:09:24.229 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:09:24.291 INFO:tasks.ceph:Compressing logs... 2024-09-22T03:09:24.292 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:09:24.358 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-22T03:09:24.358 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31725.log 2024-09-22T03:09:24.358 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32185.log 2024-09-22T03:09:24.358 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.31725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28719.log 2024-09-22T03:09:24.358 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31725.log.gz 2024-09-22T03:09:24.375 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28775.log 2024-09-22T03:09:24.375 INFO:teuthology.orchestra.run.smithi039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32185.log.gz 2024-09-22T03:09:24.375 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29677.log 2024-09-22T03:09:24.375 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28719.log.gz 2024-09-22T03:09:24.375 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29036.log 2024-09-22T03:09:24.375 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28775.log.gz 2024-09-22T03:09:24.376 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29677.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-22T03:09:24.376 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32145.log 2024-09-22T03:09:24.376 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29036.log.gz 2024-09-22T03:09:24.376 INFO:teuthology.orchestra.run.smithi039.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29677.log.gz 2024-09-22T03:09:24.376 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28301.log 2024-09-22T03:09:24.376 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29287.log 2024-09-22T03:09:24.376 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32145.log.gz 2024-09-22T03:09:24.376 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31809.log 2024-09-22T03:09:24.377 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28301.log.gz 2024-09-22T03:09:24.377 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29287.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29287.log.gzgzip 2024-09-22T03:09:24.377 INFO:teuthology.orchestra.run.smithi039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29248.log 2024-09-22T03:09:24.377 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29154.log 2024-09-22T03:09:24.377 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31809.log.gz 2024-09-22T03:09:24.377 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29248.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29248.log.gz 2024-09-22T03:09:24.377 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28550.log 2024-09-22T03:09:24.377 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28912.log 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29154.log.gz 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28550.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28550.log.gz 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29374.log 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32061.log 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28912.log.gz 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29374.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29374.log.gz 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31935.log 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32061.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-22T03:09:24.378 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32061.log.gz 2024-09-22T03:09:24.379 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32103.log 2024-09-22T03:09:24.379 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31935.log.gz 2024-09-22T03:09:24.379 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27155.log 2024-09-22T03:09:24.379 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32103.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32103.log.gz 2024-09-22T03:09:24.379 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29458.log 2024-09-22T03:09:24.379 INFO:teuthology.orchestra.run.smithi039.stderr: 87.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-22T03:09:24.379 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31893.log 2024-09-22T03:09:24.379 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.tmp-client.admin.27155.log: /var/log/ceph/ceph-client.admin.29458.log: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27155.log.gz 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31767.log 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29458.log.gz 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29034.log 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31893.log.gz 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28800.log 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31767.log.gz 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29034.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29034.log.gz 2024-09-22T03:09:24.380 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28800.log: gzip -5 66.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.28800.log.gz -- 2024-09-22T03:09:24.381 INFO:teuthology.orchestra.run.smithi039.stderr: /var/log/ceph/ceph-osd.2.log 2024-09-22T03:09:24.381 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28954.log 2024-09-22T03:09:24.381 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32019.log 2024-09-22T03:09:24.381 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28954.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-22T03:09:24.381 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28954.log.gz 2024-09-22T03:09:24.381 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29161.log 2024-09-22T03:09:24.381 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32019.log.gz 2024-09-22T03:09:24.381 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29161.log: gzip -5 --verbose -- 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29161.log.gz 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr: /var/log/ceph/ceph-client.admin.29416.log 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28103.log 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr: 86.3%/var/log/ceph/ceph-client.admin.29416.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31977.log 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29416.log.gz 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31683.log 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28103.log.gz 2024-09-22T03:09:24.382 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28191.log 2024-09-22T03:09:24.383 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31977.log.gz 2024-09-22T03:09:24.383 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.31683.log: /var/log/ceph/ceph-client.admin.28385.log 2024-09-22T03:09:24.383 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31683.log.gz 2024-09-22T03:09:24.383 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28191.log: gzip 50.4% -5 --verbose -- /var/log/ceph/ceph-client.admin.29288.log 2024-09-22T03:09:24.383 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28191.log.gz 2024-09-22T03:09:24.383 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28385.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28385.log.gz 2024-09-22T03:09:24.383 INFO:teuthology.orchestra.run.smithi039.stderr: /var/log/ceph/ceph-client.admin.29035.log 2024-09-22T03:09:24.383 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29288.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28341.log 2024-09-22T03:09:24.384 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29288.log.gz 2024-09-22T03:09:24.384 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28870.log 2024-09-22T03:09:24.384 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29035.log.gz 2024-09-22T03:09:24.384 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28341.log: /var/log/ceph/ceph-client.admin.29162.log 2024-09-22T03:09:24.384 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28341.log.gz/var/log/ceph/ceph-client.admin.28870.log: 2024-09-22T03:09:24.384 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31851.log 2024-09-22T03:09:24.384 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28870.log.gz 2024-09-22T03:09:24.384 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28577.log 2024-09-22T03:09:24.385 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29162.log: /var/log/ceph/ceph-client.admin.31851.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29162.log.gz 2024-09-22T03:09:24.385 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31851.log.gz 2024-09-22T03:09:24.385 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28577.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28577.log.gz 2024-09-22T03:09:24.431 INFO:teuthology.orchestra.run.smithi039.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-22T03:09:24.541 INFO:teuthology.orchestra.run.smithi039.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-22T03:09:25.342 INFO:teuthology.orchestra.run.smithi039.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-22T03:09:27.301 INFO:teuthology.orchestra.run.smithi039.stderr: 96.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-22T03:09:28.068 INFO:teuthology.orchestra.run.smithi039.stderr: 96.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-22T03:09:28.070 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-22T03:09:28.070 INFO:teuthology.orchestra.run.smithi039.stderr:real 0m3.723s 2024-09-22T03:09:28.070 INFO:teuthology.orchestra.run.smithi039.stderr:user 0m7.569s 2024-09-22T03:09:28.070 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m0.342s 2024-09-22T03:09:28.071 INFO:tasks.ceph:Archiving logs... 2024-09-22T03:09:28.071 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914717/remote/smithi039/log 2024-09-22T03:09:28.075 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:09:28.908 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T03:09:28.916 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T03:09:28.916 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:09:28.984 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T03:09:28.993 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T03:09:28.993 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:09:29.013 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:09:29.017 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:09:29.018 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-09-22T03:09:29.018 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv01.front.sepia.ceph.com 3 6 377 26 -34us[ -52us] +/- 50ms 2024-09-22T03:09:29.018 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 24 +161us[ +161us] +/- 54ms 2024-09-22T03:09:29.018 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -198us[ -198us] +/- 71ms 2024-09-22T03:09:29.018 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:09:29.019 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T03:09:29.055 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T03:09:29.056 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T03:09:29.071 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:09:29.135 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:09:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:09:29.163 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T03:09:29.172 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T03:09:29.189 INFO:teuthology.task.internal:Duration was 905.859593 seconds 2024-09-22T03:09:29.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T03:09:29.206 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T03:09:29.206 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:09:29.246 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:09:29.578 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T03:09:29.578 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-09-22T03:09:29.579 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:09:29.639 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T03:09:29.639 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:09:29.706 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T03:09:29.707 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:09:29.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T03:09:29.859 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T03:09:29.859 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:09:29.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T03:09:29.898 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:09:29.950 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-09-22T03:09:29.970 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:09:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:09:30.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T03:09:30.035 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T03:09:30.036 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914717/remote/smithi039 2024-09-22T03:09:30.036 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:09:30.099 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T03:09:30.099 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:09:30.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T03:09:30.159 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T03:09:30.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T03:09:30.224 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T03:09:30.224 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:09:30.240 INFO:teuthology.orchestra.run.smithi039.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 22 03:09 /home/ubuntu/cephtest 2024-09-22T03:09:30.240 INFO:teuthology.orchestra.run.smithi039.stdout: 277978 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 22 03:01 /home/ubuntu/cephtest/mnt.0 2024-09-22T03:09:30.240 INFO:teuthology.orchestra.run.smithi039.stdout: 277979 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 22 03:02 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-22T03:09:30.241 INFO:teuthology.orchestra.run.smithi039.stdout: 290501 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-22T03:09:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:09:30.242 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-22T03:09:30.242 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-22T03:09:30.248 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T03:09:30.258 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T03:09:30.274 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi039 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-22T03:09:30.277 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests_old_format} duration: 905.8595929145813 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi039 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bc51ec3968594bee92c25ac23d1331cf status: fail success: false 2024-09-22T03:09:30.278 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:09:30.368 INFO:teuthology.run:FAIL