2024-09-22T03:05:12.374 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T03:05:12.390 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:05:12.476 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914750 branch: squid description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914750' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(POOL_FULL\) sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid env: CRIMSON_COMPAT: '1' sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1158 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHZop/epIufQV+YLQUqNrG6SwAwvBD9Dz6tGqTUcor19a4MBCJCK0GhmKqlWGPGW22iu81oUCy5kTF/fnj6sGT8= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh env: RBD_FEATURES: '61' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T03:05:12.476 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-22T03:05:12.477 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-22T03:05:12.477 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T03:05:12.479 INFO:teuthology.task.internal:Checking packages... 2024-09-22T03:05:12.500 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-22T03:05:12.501 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T03:05:12.501 INFO:teuthology.packaging:ref: None 2024-09-22T03:05:12.501 INFO:teuthology.packaging:tag: None 2024-09-22T03:05:12.501 INFO:teuthology.packaging:branch: squid 2024-09-22T03:05:12.501 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:05:12.501 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-22T03:05:12.693 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-22T03:05:12.694 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T03:05:12.702 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T03:05:12.702 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T03:05:12.726 INFO:teuthology.task.internal:Saving configuration 2024-09-22T03:05:12.736 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T03:05:12.747 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T03:05:12.772 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914750', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 03:00:18.004186', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHZop/epIufQV+YLQUqNrG6SwAwvBD9Dz6tGqTUcor19a4MBCJCK0GhmKqlWGPGW22iu81oUCy5kTF/fnj6sGT8='} 2024-09-22T03:05:12.772 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T03:05:12.778 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-22T03:05:12.778 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T03:05:12.811 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f92d192c160>, signals=[15]) 2024-09-22T03:05:12.811 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T03:05:12.819 INFO:teuthology.task.internal:Opening connections... 2024-09-22T03:05:12.819 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:05:12.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:05:12.897 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T03:05:12.904 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-09-22T03:05:12.977 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-09-22T03:05:12.977 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-09-22T03:05:13.035 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T03:05:13.036 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-09-22T03:05:13.036 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T03:05:13.036 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T03:05:13.036 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T03:05:13.036 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-09-22T03:05:13.055 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T03:05:13.064 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T03:05:13.069 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T03:05:13.069 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-09-22T03:05:13.091 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T03:05:13.097 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T03:05:13.098 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-09-22T03:05:13.156 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T03:05:13.183 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T03:05:13.196 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T03:05:13.196 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T03:05:13.197 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T03:05:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:05:13.213 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-09-22T03:05:13.271 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-503.el9.x86_64 2024-09-22T03:05:13.271 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-503.el9.x86_64 2024-09-22T03:05:13.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-09-22T03:05:16.768 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-22T03:05:24.768 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 20 MB 00:06 2024-09-22T03:05:30.226 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-22T03:05:31.768 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 19 kB 00:00 2024-09-22T03:05:32.830 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-22T03:05:38.227 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-22T03:05:40.125 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T03:05:40.126 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T03:05:40.232 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:05:40.239 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-09-22T03:05:40.240 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:05:40.241 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 79 M 2024-09-22T03:05:40.241 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 126 M 2024-09-22T03:05:40.241 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:05:40.629 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 331 kB/s | 36 kB 00:00 2024-09-22T03:05:46.970 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.4 MB/s | 16 MB 00:06 2024-09-22T03:05:53.702 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.2 MB/s | 29 MB 00:13 2024-09-22T03:05:56.918 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.1 MB/s | 35 MB 00:16 2024-09-22T03:05:56.919 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:05:56.919 INFO:teuthology.orchestra.run.smithi070.stdout:Total 4.7 MB/s | 79 MB 00:16 2024-09-22T03:05:57.603 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:05:57.971 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:05:57.971 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:05:58.930 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:05:58.930 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:06:01.384 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:06:02.145 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:06:02.227 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:06:04.971 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:06:05.114 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:06:09.950 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:06:09.997 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:06:14.843 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:06:52.917 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:06:52.956 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:06:53.304 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:06:53.304 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:06:53.304 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:06:53.304 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:06:54.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:06:54.103 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:06:54.103 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:06:54.104 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T03:06:54.104 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T03:06:54.104 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T03:06:54.104 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T03:06:54.104 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:06:54.104 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:06:54.433 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:06:54.519 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:06:54.520 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:06:54.520 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T03:06:54.520 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T03:06:54.520 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T03:06:54.520 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T03:06:54.520 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T03:06:54.520 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:06:54.579 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:06:54.579 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:06:54.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.06 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 82.el9 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Group : Unspecified 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 8274670 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T03:06:54.619 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : builder@centos.org 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T03:06:54.620 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T03:06:54.621 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T03:06:54.621 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-09-22T03:06:54.621 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:06:54.621 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T03:06:54.622 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T03:06:54.622 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 5.14.0-508.el9.x86_64 2024-09-22T03:06:54.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T03:06:55.296 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-09-22T03:06:56.640 INFO:teuthology.orchestra.run.smithi070.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T03:06:56.668 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-09-22T03:06:56.671 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-09-22T03:06:56.687 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.ORE1y3T9XM 2024-09-22T03:06:56.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ORE1y3T9XM 2024-09-22T03:06:56.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.ORE1y3T9XM 2024-09-22T03:06:56.891 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.ORE1y3T9XM is 6KB 2024-09-22T03:06:56.913 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.ORE1y3T9XM 2024-09-22T03:06:56.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T03:06:56.991 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T03:06:56.991 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T03:06:56.991 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T03:06:56.991 INFO:teuthology.orchestra.run.smithi070.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T03:06:56.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T03:06:57.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-09-22T03:07:27.185 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T03:07:27.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-22T03:07:27.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:08:27.243 DEBUG:teuthology.orchestra.remote:timed out 2024-09-22T03:08:36.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-22T03:08:36.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:08:36.693 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-09-22T03:08:37.108 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-22T03:08:37.109 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:08:37.109 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-09-22T03:08:37.123 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:08:37.124 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:08:37.124 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:08:37.124 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:08:38.125 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T03:08:38.125 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:08:38.204 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-09-22T03:08:38.227 DEBUG:teuthology.parallel:result is None 2024-09-22T03:08:38.227 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T03:08:38.237 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T03:08:38.237 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:08:38.252 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T03:08:38.258 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T03:08:38.270 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T03:08:38.271 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:08:38.311 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T03:08:38.317 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T03:08:38.317 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:08:38.370 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:08:38.380 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:08:38.381 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T03:08:38.388 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T03:08:38.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:08:38.444 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T03:08:38.452 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T03:08:38.452 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:08:38.496 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:08:38.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:08:38.631 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:08:38.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:08:38.708 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:08:38.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:08:38.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-09-22T03:08:38.853 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:08:39.006 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T03:08:39.013 INFO:teuthology.task.internal:Starting timer... 2024-09-22T03:08:39.013 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T03:08:39.023 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T03:08:39.031 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-22T03:08:39.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-09-22T03:08:39.139 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-09-22T03:08:39.141 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T03:08:39.141 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-09-22T03:08:39.159 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-09-22T03:08:39.160 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive'} 2024-09-22T03:08:39.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:08:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:08:39.222 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T03:08:39.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:08:39.288 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T03:08:39.301 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T03:08:39.316 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T03:08:39.317 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T03:12:26.890 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com')] 2024-09-22T03:12:26.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-22T03:12:26.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:12:26.969 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-09-22T03:12:27.049 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-22T03:12:27.049 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T03:12:27.060 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T03:12:27.060 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T03:12:27.061 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:12:27.118 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T03:12:27.132 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T03:12:27.156 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-09-22T03:12:27.167 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-09-22T03:12:27.181 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T03:12:27.195 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T03:12:27.251 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:12:27.254 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:12:27.254 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-09-22T03:12:27.254 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:12:27.254 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:12:27.254 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:12:27.255 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:12:27.255 INFO:teuthology.run_tasks:Running task install... 2024-09-22T03:12:27.263 DEBUG:teuthology.task.install:project ceph 2024-09-22T03:12:27.263 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-22T03:12:27.263 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'} 2024-09-22T03:12:27.263 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T03:12:27.269 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T03:12:27.269 INFO:teuthology.task.install:extra packages: [] 2024-09-22T03:12:27.269 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-22T03:12:27.270 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:12:27.407 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/ 2024-09-22T03:12:27.407 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-22T03:12:27.515 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T03:12:27.515 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:12:27.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T03:12:27.542 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T03:12:27.542 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T03:12:27.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T03:12:27.683 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T03:12:27.748 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-09-22T03:12:27.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-09-22T03:12:27.971 INFO:teuthology.orchestra.run.smithi070.stdout:45 files removed 2024-09-22T03:12:27.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw 2024-09-22T03:12:28.563 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 310 kB/s | 87 kB 00:00 2024-09-22T03:12:28.847 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-22T03:12:29.122 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 8.3 kB/s | 2.0 kB 00:00 2024-09-22T03:12:30.064 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.3 MB 00:00 2024-09-22T03:12:31.907 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-22T03:12:37.674 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.5 MB 00:02 2024-09-22T03:12:39.891 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 19 kB 00:00 2024-09-22T03:12:41.087 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-22T03:12:46.433 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-09-22T03:12:48.643 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 9.9 M 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:12:48.669 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 19 M 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 760 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 501 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T03:12:48.670 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.1 M 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 161 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 301 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T03:12:48.671 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout:Install 30 Packages 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 2 Packages 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:12:48.672 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 58 M 2024-09-22T03:12:48.673 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:12:50.251 INFO:teuthology.orchestra.run.smithi070.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 4.4 MB/s | 5.4 MB 00:01 2024-09-22T03:12:50.345 INFO:teuthology.orchestra.run.smithi070.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 268 kB/s | 25 kB 00:00 2024-09-22T03:12:50.629 INFO:teuthology.orchestra.run.smithi070.stdout:(3/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 2.6 MB/s | 760 kB 00:00 2024-09-22T03:12:50.779 INFO:teuthology.orchestra.run.smithi070.stdout:(4/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 3.3 MB/s | 501 kB 00:00 2024-09-22T03:12:51.495 INFO:teuthology.orchestra.run.smithi070.stdout:(5/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 4.0 MB/s | 9.9 MB 00:02 2024-09-22T03:12:51.588 INFO:teuthology.orchestra.run.smithi070.stdout:(6/32): python3-ceph-argparse-19.1.1-373.g4e45e 493 kB/s | 45 kB 00:00 2024-09-22T03:12:51.713 INFO:teuthology.orchestra.run.smithi070.stdout:(7/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_64 5.4 MB/s | 5.1 MB 00:00 2024-09-22T03:12:51.746 INFO:teuthology.orchestra.run.smithi070.stdout:(8/32): python3-ceph-common-19.1.1-373.g4e45e6b 858 kB/s | 136 kB 00:00 2024-09-22T03:12:51.797 INFO:teuthology.orchestra.run.smithi070.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 1.9 MB/s | 161 kB 00:00 2024-09-22T03:12:51.839 INFO:teuthology.orchestra.run.smithi070.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 3.4 MB/s | 321 kB 00:00 2024-09-22T03:12:51.889 INFO:teuthology.orchestra.run.smithi070.stdout:(11/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 3.2 MB/s | 301 kB 00:00 2024-09-22T03:12:51.931 INFO:teuthology.orchestra.run.smithi070.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.1 MB/s | 99 kB 00:00 2024-09-22T03:12:52.023 INFO:teuthology.orchestra.run.smithi070.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 799 kB/s | 107 kB 00:00 2024-09-22T03:12:52.056 INFO:teuthology.orchestra.run.smithi070.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-22T03:12:52.082 INFO:teuthology.orchestra.run.smithi070.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 771 kB/s | 45 kB 00:00 2024-09-22T03:12:52.157 INFO:teuthology.orchestra.run.smithi070.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-09-22T03:12:52.182 INFO:teuthology.orchestra.run.smithi070.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-22T03:12:52.199 INFO:teuthology.orchestra.run.smithi070.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-22T03:12:52.224 INFO:teuthology.orchestra.run.smithi070.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 221 kB 00:00 2024-09-22T03:12:52.241 INFO:teuthology.orchestra.run.smithi070.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 853 kB/s | 36 kB 00:00 2024-09-22T03:12:52.258 INFO:teuthology.orchestra.run.smithi070.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-22T03:12:52.309 INFO:teuthology.orchestra.run.smithi070.stdout:(22/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.5 MB/s | 308 kB 00:00 2024-09-22T03:12:52.334 INFO:teuthology.orchestra.run.smithi070.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-22T03:12:52.360 INFO:teuthology.orchestra.run.smithi070.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-22T03:12:52.385 INFO:teuthology.orchestra.run.smithi070.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-22T03:12:52.452 INFO:teuthology.orchestra.run.smithi070.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-22T03:12:52.485 INFO:teuthology.orchestra.run.smithi070.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-22T03:12:52.628 INFO:teuthology.orchestra.run.smithi070.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-22T03:12:52.670 INFO:teuthology.orchestra.run.smithi070.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-09-22T03:12:53.028 INFO:teuthology.orchestra.run.smithi070.stdout:(30/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x 4.9 MB/s | 19 MB 00:04 2024-09-22T03:12:53.129 INFO:teuthology.orchestra.run.smithi070.stdout:(31/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 6.4 MB/s | 3.2 MB 00:00 2024-09-22T03:12:53.204 INFO:teuthology.orchestra.run.smithi070.stdout:(32/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 5.8 MB/s | 3.1 MB 00:00 2024-09-22T03:12:53.208 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:12:53.209 INFO:teuthology.orchestra.run.smithi070.stdout:Total 13 MB/s | 58 MB 00:04 2024-09-22T03:12:53.306 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:12:53.341 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:12:53.341 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:12:54.020 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:12:54.020 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:12:55.489 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:12:55.871 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-22T03:12:55.980 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:12:56.073 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:12:56.163 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:12:56.199 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:12:56.275 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:12:56.311 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-09-22T03:12:56.345 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-09-22T03:12:56.367 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:12:56.388 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:12:56.562 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:12:56.590 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:12:56.619 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-09-22T03:12:56.663 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 10/34 2024-09-22T03:12:56.692 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:12:56.758 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-09-22T03:12:56.781 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:12:56.852 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:12:56.888 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-09-22T03:12:57.011 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-22T03:12:57.039 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-22T03:12:57.080 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-22T03:12:57.130 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-22T03:12:57.167 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-22T03:12:57.200 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-22T03:12:57.519 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-22T03:12:57.562 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:12:57.682 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:12:57.892 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-22T03:12:57.928 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-22T03:12:58.434 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-22T03:12:58.473 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:12:58.528 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:12:58.607 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-22T03:12:58.634 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-22T03:13:00.896 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:13:00.947 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:13:01.511 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:13:01.551 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:13:01.628 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:13:01.628 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T03:13:01.628 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T03:13:01.629 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:01.677 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:13:11.655 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:13:11.655 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-09-22T03:13:11.655 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-09-22T03:13:11.655 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-09-22T03:13:11.655 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-09-22T03:13:11.655 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-09-22T03:13:11.656 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-09-22T03:13:11.656 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-09-22T03:13:11.656 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:12.656 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:13:12.684 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:13:12.685 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:13:12.685 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T03:13:12.685 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:13:12.685 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:13:12.685 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:12.686 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:13:12.766 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:13:12.767 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:13:13.949 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:13:13.949 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-22T03:13:13.949 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:13:13.949 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:13:13.949 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:13:13.949 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-22T03:13:13.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-22T03:13:13.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-22T03:13:13.952 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-22T03:13:13.952 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:13:13.952 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-22T03:13:13.952 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-22T03:13:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T03:13:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T03:13:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T03:13:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T03:13:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:15.086 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:13:15.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-test 2024-09-22T03:13:16.029 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:30 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:13:16.166 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:13:16.167 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:13:16.167 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:13:16.167 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:13:16.167 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 124 M 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: libasan x86_64 11.5.0-2.el9 appstream 414 k 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 158 k 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout:Install 7 Packages 2024-09-22T03:13:16.168 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:16.169 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 125 M 2024-09-22T03:13:16.169 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 1.1 G 2024-09-22T03:13:16.169 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:13:16.439 INFO:teuthology.orchestra.run.smithi070.stdout:(1/7): libasan-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 414 kB 00:00 2024-09-22T03:13:16.480 INFO:teuthology.orchestra.run.smithi070.stdout:(2/7): libubsan-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 183 kB 00:00 2024-09-22T03:13:16.506 INFO:teuthology.orchestra.run.smithi070.stdout:(3/7): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 690 kB/s | 158 kB 00:00 2024-09-22T03:13:16.532 INFO:teuthology.orchestra.run.smithi070.stdout:(4/7): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-09-22T03:13:16.573 INFO:teuthology.orchestra.run.smithi070.stdout:(5/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.9 MB/s | 122 kB 00:00 2024-09-22T03:13:16.615 INFO:teuthology.orchestra.run.smithi070.stdout:(6/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 585 kB/s | 64 kB 00:00 2024-09-22T03:13:20.607 INFO:teuthology.orchestra.run.smithi070.stdout:(7/7): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 29 MB/s | 124 MB 00:04 2024-09-22T03:13:20.609 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:13:20.609 INFO:teuthology.orchestra.run.smithi070.stdout:Total 28 MB/s | 125 MB 00:04 2024-09-22T03:13:20.664 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:13:20.684 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:13:20.684 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:13:21.503 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:13:21.503 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:13:22.561 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:13:22.630 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-09-22T03:13:22.682 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-09-22T03:13:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-09-22T03:13:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:13:22.808 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libasan-11.5.0-2.el9.x86_64 5/7 2024-09-22T03:13:22.834 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:13:39.996 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:13:40.073 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:13:48.297 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:13:48.297 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/7 2024-09-22T03:13:48.297 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/7 2024-09-22T03:13:48.297 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libasan-11.5.0-2.el9.x86_64 3/7 2024-09-22T03:13:48.297 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:13:48.297 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-09-22T03:13:48.298 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-09-22T03:13:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-09-22T03:13:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:48.724 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:13:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: libasan-11.5.0-2.el9.x86_64 2024-09-22T03:13:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:13:48.724 INFO:teuthology.orchestra.run.smithi070.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T03:13:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T03:13:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T03:13:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T03:13:48.725 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:48.725 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:13:49.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph 2024-09-22T03:13:49.608 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:03 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:13:49.755 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.3 M 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.9 M 2024-09-22T03:13:49.758 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 120 M 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-22T03:13:49.759 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T03:13:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout:Install 38 Packages 2024-09-22T03:13:49.761 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:13:49.762 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 134 M 2024-09-22T03:13:49.762 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 1.4 G 2024-09-22T03:13:49.762 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:13:50.310 INFO:teuthology.orchestra.run.smithi070.stdout:(1/38): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 57 kB/s | 6.6 kB 00:00 2024-09-22T03:13:50.518 INFO:teuthology.orchestra.run.smithi070.stdout:(2/38): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 7.0 MB/s | 2.3 MB 00:00 2024-09-22T03:13:50.560 INFO:teuthology.orchestra.run.smithi070.stdout:(3/38): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 4.1 MB/s | 1.5 MB 00:00 2024-09-22T03:13:50.653 INFO:teuthology.orchestra.run.smithi070.stdout:(4/38): ceph-mgr-modules-core-19.1.1-373.g4e45e 2.7 MB/s | 249 kB 00:00 2024-09-22T03:13:50.720 INFO:teuthology.orchestra.run.smithi070.stdout:(5/38): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-09-22T03:13:51.403 INFO:teuthology.orchestra.run.smithi070.stdout:(6/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 337 kB/s | 253 kB 00:00 2024-09-22T03:13:51.745 INFO:teuthology.orchestra.run.smithi070.stdout:(7/38): python3-cryptography-36.0.1-4.el9.x86_6 1.2 MB/s | 1.2 MB 00:01 2024-09-22T03:13:52.104 INFO:teuthology.orchestra.run.smithi070.stdout:(8/38): python3-ply-3.11-14.el9.noarch.rpm 152 kB/s | 106 kB 00:00 2024-09-22T03:13:52.487 INFO:teuthology.orchestra.run.smithi070.stdout:(9/38): python3-pycparser-2.20-6.el9.noarch.rpm 182 kB/s | 135 kB 00:00 2024-09-22T03:13:53.688 INFO:teuthology.orchestra.run.smithi070.stdout:(10/38): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 38 MB/s | 120 MB 00:03 2024-09-22T03:13:53.721 INFO:teuthology.orchestra.run.smithi070.stdout:(11/38): python3-requests-2.25.1-8.el9.noarch.r 77 kB/s | 125 kB 00:01 2024-09-22T03:13:54.263 INFO:teuthology.orchestra.run.smithi070.stdout:(12/38): python3-urllib3-1.26.5-6.el9.noarch.rp 121 kB/s | 215 kB 00:01 2024-09-22T03:13:54.488 INFO:teuthology.orchestra.run.smithi070.stdout:(13/38): lua-5.4.4-4.el9.x86_64.rpm 238 kB/s | 188 kB 00:00 2024-09-22T03:13:54.731 INFO:teuthology.orchestra.run.smithi070.stdout:(14/38): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-09-22T03:13:55.047 INFO:teuthology.orchestra.run.smithi070.stdout:(15/38): python3-mako-1.1.4-6.el9.noarch.rpm 220 kB/s | 172 kB 00:00 2024-09-22T03:13:55.081 INFO:teuthology.orchestra.run.smithi070.stdout:(16/38): python3-markupsafe-1.1.1-12.el9.x86_64 59 kB/s | 35 kB 00:00 2024-09-22T03:13:55.389 INFO:teuthology.orchestra.run.smithi070.stdout:(17/38): python3-packaging-20.9-5.el9.noarch.rp 119 kB/s | 77 kB 00:00 2024-09-22T03:13:55.664 INFO:teuthology.orchestra.run.smithi070.stdout:(18/38): python3-toml-0.10.2-6.el9.noarch.rpm 71 kB/s | 42 kB 00:00 2024-09-22T03:13:55.690 INFO:teuthology.orchestra.run.smithi070.stdout:(19/38): python3-pytz-2021.1-5.el9.noarch.rpm 79 kB/s | 51 kB 00:00 2024-09-22T03:13:55.740 INFO:teuthology.orchestra.run.smithi070.stdout:(20/38): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-22T03:13:55.765 INFO:teuthology.orchestra.run.smithi070.stdout:(21/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.2 MB/s | 172 kB 00:00 2024-09-22T03:13:55.790 INFO:teuthology.orchestra.run.smithi070.stdout:(22/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-09-22T03:13:55.815 INFO:teuthology.orchestra.run.smithi070.stdout:(23/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-22T03:13:55.841 INFO:teuthology.orchestra.run.smithi070.stdout:(24/38): python3-jaraco-classes-3.2.1-5.el9.noa 708 kB/s | 18 kB 00:00 2024-09-22T03:13:55.866 INFO:teuthology.orchestra.run.smithi070.stdout:(25/38): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-22T03:13:55.892 INFO:teuthology.orchestra.run.smithi070.stdout:(26/38): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-22T03:13:55.917 INFO:teuthology.orchestra.run.smithi070.stdout:(27/38): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-22T03:13:55.942 INFO:teuthology.orchestra.run.smithi070.stdout:(28/38): lua-devel-5.4.4-4.el9.x86_64.rpm 40 kB/s | 22 kB 00:00 2024-09-22T03:13:55.967 INFO:teuthology.orchestra.run.smithi070.stdout:(29/38): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-22T03:13:56.001 INFO:teuthology.orchestra.run.smithi070.stdout:(30/38): python3-logutils-0.3.5-21.el9.noarch.r 559 kB/s | 46 kB 00:00 2024-09-22T03:13:56.026 INFO:teuthology.orchestra.run.smithi070.stdout:(31/38): python3-portend-3.1.0-2.el9.noarch.rpm 706 kB/s | 16 kB 00:00 2024-09-22T03:13:56.051 INFO:teuthology.orchestra.run.smithi070.stdout:(32/38): python3-more-itertools-8.12.0-2.el9.no 723 kB/s | 79 kB 00:00 2024-09-22T03:13:56.077 INFO:teuthology.orchestra.run.smithi070.stdout:(33/38): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-22T03:13:56.102 INFO:teuthology.orchestra.run.smithi070.stdout:(34/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-22T03:13:56.128 INFO:teuthology.orchestra.run.smithi070.stdout:(35/38): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-22T03:13:56.161 INFO:teuthology.orchestra.run.smithi070.stdout:(36/38): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-22T03:13:56.187 INFO:teuthology.orchestra.run.smithi070.stdout:(37/38): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-09-22T03:13:56.220 INFO:teuthology.orchestra.run.smithi070.stdout:(38/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-22T03:13:56.226 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:13:56.226 INFO:teuthology.orchestra.run.smithi070.stdout:Total 21 MB/s | 134 MB 00:06 2024-09-22T03:13:56.393 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:13:56.433 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:13:56.434 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:13:57.533 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:13:57.533 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:13:59.525 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:13:59.589 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-09-22T03:13:59.633 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-09-22T03:13:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-09-22T03:13:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-09-22T03:13:59.782 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-09-22T03:13:59.815 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-09-22T03:13:59.915 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-09-22T03:13:59.997 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-09-22T03:14:00.046 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/38 2024-09-22T03:14:00.089 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-09-22T03:14:00.119 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-09-22T03:14:00.154 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-09-22T03:14:00.187 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-09-22T03:14:00.228 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-09-22T03:14:00.271 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-09-22T03:14:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-09-22T03:14:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-09-22T03:14:00.386 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-09-22T03:14:00.460 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-09-22T03:14:00.607 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-09-22T03:14:00.667 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-09-22T03:14:19.534 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-09-22T03:14:19.563 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:14:19.591 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:14:19.591 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:14:19.591 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:14:19.591 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:14:19.591 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:14:19.591 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:19.687 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-09-22T03:14:19.726 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-09-22T03:14:19.774 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-09-22T03:14:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-09-22T03:14:20.024 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-09-22T03:14:20.090 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-09-22T03:14:20.166 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-09-22T03:14:20.255 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-09-22T03:14:20.288 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-09-22T03:14:20.411 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-09-22T03:14:20.579 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 34/38 2024-09-22T03:14:20.607 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:14:20.631 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:14:20.631 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:14:20.631 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:14:20.631 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:14:20.632 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:14:20.632 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:21.100 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:14:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:14:21.123 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:14:21.123 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:14:21.123 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:14:21.123 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:14:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:21.350 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:14:21.374 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:14:21.375 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:21.440 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:14:31.652 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:14:31.652 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/38 2024-09-22T03:14:31.652 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/38 2024-09-22T03:14:31.652 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/38 2024-09-22T03:14:31.652 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/38 2024-09-22T03:14:31.652 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-09-22T03:14:31.653 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-09-22T03:14:31.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-09-22T03:14:31.655 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-09-22T03:14:31.655 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-09-22T03:14:31.655 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-09-22T03:14:31.655 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-09-22T03:14:31.655 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 36/38 2024-09-22T03:14:31.655 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:14:32.571 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:14:32.572 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:14:32.573 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:14:32.574 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:14:32.574 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:32.574 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:14:32.831 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-base 2024-09-22T03:14:33.412 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:47 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:14:33.503 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:14:33.546 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:14:33.547 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:14:33.547 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:14:33.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install cephadm 2024-09-22T03:14:34.182 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:48 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:14:34.315 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-22T03:14:34.316 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:34.317 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 765 k 2024-09-22T03:14:34.317 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 771 k 2024-09-22T03:14:34.317 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:14:34.601 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-22T03:14:34.601 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:14:34.601 INFO:teuthology.orchestra.run.smithi070.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-22T03:14:34.601 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:14:34.607 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:14:34.607 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:14:34.613 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:14:34.613 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:14:34.684 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:14:35.111 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:14:35.204 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:14:35.571 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:14:35.822 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:14:35.822 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:35.822 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:14:35.822 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:14:35.822 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:35.822 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:14:36.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:14:36.637 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:50 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:14:36.771 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:14:36.772 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:36.772 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-22T03:14:36.772 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:36.772 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:14:36.772 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 147 k 2024-09-22T03:14:36.772 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:36.772 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:14:36.773 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:36.773 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-22T03:14:36.773 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:36.773 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 147 k 2024-09-22T03:14:36.773 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 481 k 2024-09-22T03:14:36.773 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:14:37.140 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 401 kB/s | 147 kB 00:00 2024-09-22T03:14:37.141 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:14:37.141 INFO:teuthology.orchestra.run.smithi070.stdout:Total 400 kB/s | 147 kB 00:00 2024-09-22T03:14:37.141 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:14:37.149 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:14:37.149 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:14:37.205 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:14:37.206 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:14:37.344 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:14:37.408 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:14:37.433 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:14:37.433 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:14:37.433 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:14:37.433 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:38.334 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:14:38.334 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:38.334 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:14:38.334 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:14:38.334 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:38.334 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:14:38.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr 2024-09-22T03:14:39.199 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:53 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:14:39.289 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:14:39.332 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:14:39.333 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:14:39.333 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:14:39.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:14:39.977 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:53 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:14:40.109 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:14:40.110 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:40.110 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:14:40.110 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:40.110 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:14:40.110 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-22T03:14:40.110 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:14:40.110 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:14:40.110 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout:Install 11 Packages 2024-09-22T03:14:40.111 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:40.112 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 7.7 M 2024-09-22T03:14:40.112 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 101 M 2024-09-22T03:14:40.112 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:14:40.519 INFO:teuthology.orchestra.run.smithi070.stdout:(1/11): ceph-prometheus-alerts-19.1.1-373.g4e45 60 kB/s | 16 kB 00:00 2024-09-22T03:14:40.545 INFO:teuthology.orchestra.run.smithi070.stdout:(2/11): ceph-grafana-dashboards-19.1.1-373.g4e4 105 kB/s | 32 kB 00:00 2024-09-22T03:14:40.778 INFO:teuthology.orchestra.run.smithi070.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-09-22T03:14:40.870 INFO:teuthology.orchestra.run.smithi070.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.8 MB/s | 548 kB 00:00 2024-09-22T03:14:40.896 INFO:teuthology.orchestra.run.smithi070.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 773 kB/s | 19 kB 00:00 2024-09-22T03:14:41.012 INFO:teuthology.orchestra.run.smithi070.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 18 MB/s | 2.0 MB 00:00 2024-09-22T03:14:41.038 INFO:teuthology.orchestra.run.smithi070.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-22T03:14:41.055 INFO:teuthology.orchestra.run.smithi070.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-22T03:14:41.080 INFO:teuthology.orchestra.run.smithi070.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-22T03:14:41.322 INFO:teuthology.orchestra.run.smithi070.stdout:(10/11): ceph-mgr-dashboard-19.1.1-373.g4e45e6b 3.3 MB/s | 3.6 MB 00:01 2024-09-22T03:14:41.372 INFO:teuthology.orchestra.run.smithi070.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-09-22T03:14:41.375 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:14:41.375 INFO:teuthology.orchestra.run.smithi070.stdout:Total 6.1 MB/s | 7.7 MB 00:01 2024-09-22T03:14:41.405 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:14:41.422 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:14:41.422 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:14:41.619 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:14:41.619 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:14:42.106 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:14:42.195 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:14:42.234 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:14:42.345 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:14:42.469 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:14:42.520 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:14:42.604 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:14:42.656 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:14:42.693 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:14:42.725 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 9/11 2024-09-22T03:14:43.776 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 10/11 2024-09-22T03:14:43.809 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:14:45.113 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:14:45.113 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:14:45.114 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:14:45.437 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:14:45.438 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:14:45.438 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:14:45.438 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:45.438 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:14:45.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:14:46.263 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:00 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:14:46.397 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:14:46.398 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-22T03:14:46.398 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:14:46.398 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-22T03:14:46.398 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:14:46.399 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:14:46.400 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:46.400 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:14:46.400 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-09-22T03:14:46.400 INFO:teuthology.orchestra.run.smithi070.stdout:Install 12 Packages 2024-09-22T03:14:46.400 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:46.400 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 38 M 2024-09-22T03:14:46.400 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 212 M 2024-09-22T03:14:46.400 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:14:47.363 INFO:teuthology.orchestra.run.smithi070.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 338 kB/s | 191 kB 00:00 2024-09-22T03:14:47.439 INFO:teuthology.orchestra.run.smithi070.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 436 kB/s | 33 kB 00:00 2024-09-22T03:14:47.564 INFO:teuthology.orchestra.run.smithi070.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 801 kB 00:00 2024-09-22T03:14:47.622 INFO:teuthology.orchestra.run.smithi070.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-09-22T03:14:47.672 INFO:teuthology.orchestra.run.smithi070.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 771 kB/s | 38 kB 00:00 2024-09-22T03:14:47.718 INFO:teuthology.orchestra.run.smithi070.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 118 kB/s | 18 kB 00:00 2024-09-22T03:14:47.848 INFO:teuthology.orchestra.run.smithi070.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 28 MB/s | 4.9 MB 00:00 2024-09-22T03:14:47.965 INFO:teuthology.orchestra.run.smithi070.stdout:(8/12): ceph-mgr-diskprediction-local-19.1.1-37 6.4 MB/s | 7.4 MB 00:01 2024-09-22T03:14:47.991 INFO:teuthology.orchestra.run.smithi070.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 899 kB/s | 245 kB 00:00 2024-09-22T03:14:48.091 INFO:teuthology.orchestra.run.smithi070.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-09-22T03:14:48.174 INFO:teuthology.orchestra.run.smithi070.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.2 MB/s | 250 kB 00:00 2024-09-22T03:14:49.099 INFO:teuthology.orchestra.run.smithi070.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-09-22T03:14:49.100 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:14:49.100 INFO:teuthology.orchestra.run.smithi070.stdout:Total 14 MB/s | 38 MB 00:02 2024-09-22T03:14:49.452 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:14:49.479 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:14:49.479 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:14:49.900 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:14:49.900 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:14:50.669 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:14:50.733 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:14:50.772 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:14:50.876 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:14:50.915 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:14:51.099 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:14:51.139 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:14:51.216 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:14:51.881 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:14:52.003 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:14:53.251 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:14:54.291 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:14:54.314 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:14:56.958 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:14:56.958 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:14:56.959 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:14:57.348 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:14:57.348 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:57.348 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:57.349 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:14:57.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:14:58.345 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:12 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:14:58.479 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:14:58.480 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:58.480 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:14:58.480 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:58.480 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:14:58.480 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-22T03:14:58.480 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:14:58.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:14:58.481 INFO:teuthology.orchestra.run.smithi070.stdout:Install 13 Packages 2024-09-22T03:14:58.482 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:14:58.482 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 2.2 M 2024-09-22T03:14:58.482 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 26 M 2024-09-22T03:14:58.482 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:14:58.807 INFO:teuthology.orchestra.run.smithi070.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 311 kB/s | 26 kB 00:00 2024-09-22T03:14:58.832 INFO:teuthology.orchestra.run.smithi070.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 178 kB/s | 19 kB 00:00 2024-09-22T03:14:58.908 INFO:teuthology.orchestra.run.smithi070.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-22T03:14:58.933 INFO:teuthology.orchestra.run.smithi070.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-22T03:14:58.950 INFO:teuthology.orchestra.run.smithi070.stdout:(5/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 219 kB/s | 49 kB 00:00 2024-09-22T03:14:58.966 INFO:teuthology.orchestra.run.smithi070.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.6 MB/s | 54 kB 00:00 2024-09-22T03:14:58.992 INFO:teuthology.orchestra.run.smithi070.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-09-22T03:14:59.034 INFO:teuthology.orchestra.run.smithi070.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 384 kB/s | 32 kB 00:00 2024-09-22T03:14:59.059 INFO:teuthology.orchestra.run.smithi070.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 153 kB/s | 14 kB 00:00 2024-09-22T03:14:59.076 INFO:teuthology.orchestra.run.smithi070.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-09-22T03:14:59.101 INFO:teuthology.orchestra.run.smithi070.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.2 MB/s | 220 kB 00:00 2024-09-22T03:14:59.127 INFO:teuthology.orchestra.run.smithi070.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-22T03:14:59.160 INFO:teuthology.orchestra.run.smithi070.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.2 MB/s | 1.0 MB 00:00 2024-09-22T03:14:59.163 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:14:59.163 INFO:teuthology.orchestra.run.smithi070.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-22T03:14:59.227 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:14:59.242 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:14:59.242 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:14:59.330 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:14:59.331 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:14:59.833 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:14:59.999 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:15:00.069 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:15:00.125 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:15:00.163 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:15:00.204 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:15:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:15:00.406 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:15:00.464 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:15:01.189 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:15:01.236 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:15:01.270 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:15:01.318 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:15:01.340 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:15:02.121 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:15:02.121 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-22T03:15:02.121 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:15:02.122 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:15:02.122 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:15:02.122 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:15:02.122 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:15:02.122 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:15:02.122 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:15:02.122 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:15:02.123 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:15:02.123 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:15:02.123 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:15:02.441 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:15:02.442 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:15:02.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:15:02.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:15:02.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:15:02.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:15:02.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:15:02.443 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:02.443 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:02.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:15:03.297 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:17 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:03.430 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:03.430 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:15:03.431 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:03.432 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:15:03.432 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:03.432 INFO:teuthology.orchestra.run.smithi070.stdout:Install 6 Packages 2024-09-22T03:15:03.432 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:03.432 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 7.0 M 2024-09-22T03:15:03.432 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 33 M 2024-09-22T03:15:03.432 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:15:03.863 INFO:teuthology.orchestra.run.smithi070.stdout:(1/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 711 kB/s | 146 kB 00:00 2024-09-22T03:15:03.930 INFO:teuthology.orchestra.run.smithi070.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-22T03:15:03.955 INFO:teuthology.orchestra.run.smithi070.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-22T03:15:03.980 INFO:teuthology.orchestra.run.smithi070.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.0 MB/s | 74 kB 00:00 2024-09-22T03:15:04.414 INFO:teuthology.orchestra.run.smithi070.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 328 kB/s | 248 kB 00:00 2024-09-22T03:15:04.906 INFO:teuthology.orchestra.run.smithi070.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-09-22T03:15:04.908 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:15:04.908 INFO:teuthology.orchestra.run.smithi070.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-09-22T03:15:04.982 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:15:04.993 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:15:04.994 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:15:05.095 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:15:05.096 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:15:05.410 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:15:05.581 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:15:05.633 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:15:06.170 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:15:06.260 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:15:06.332 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:15:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:15:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:15:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-22T03:15:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:15:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:15:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:15:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:15:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:15:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:07.450 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:15:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:15:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:15:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:15:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:15:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:15:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:07.451 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:07.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-fuse 2024-09-22T03:15:08.259 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:22 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:08.393 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:08.393 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:08.393 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:15:08.393 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:08.393 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:15:08.394 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 868 k 2024-09-22T03:15:08.394 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-09-22T03:15:08.394 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:15:08.394 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:08.394 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:15:08.394 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:08.394 INFO:teuthology.orchestra.run.smithi070.stdout:Install 2 Packages 2024-09-22T03:15:08.394 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:08.395 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 948 k 2024-09-22T03:15:08.395 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 3.1 M 2024-09-22T03:15:08.395 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:15:08.817 INFO:teuthology.orchestra.run.smithi070.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 563 kB/s | 80 kB 00:00 2024-09-22T03:15:08.942 INFO:teuthology.orchestra.run.smithi070.stdout:(2/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 3.2 MB/s | 868 kB 00:00 2024-09-22T03:15:08.942 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:15:08.942 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.7 MB/s | 948 kB 00:00 2024-09-22T03:15:08.954 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:15:08.976 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:15:08.976 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:15:09.055 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:15:09.056 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:15:09.239 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:15:09.370 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:15:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:15:10.133 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:15:10.133 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-22T03:15:10.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:15:10.410 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:10.410 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:10.410 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:15:10.410 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:10.410 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:10.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-volume 2024-09-22T03:15:11.287 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:25 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:11.427 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:11.427 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 285 k 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 1.3 M 2024-09-22T03:15:11.428 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:15:11.855 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 669 kB/s | 285 kB 00:00 2024-09-22T03:15:11.856 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:15:11.856 INFO:teuthology.orchestra.run.smithi070.stdout:Total 666 kB/s | 285 kB 00:00 2024-09-22T03:15:11.856 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:15:11.866 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:15:11.866 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:15:11.894 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:15:11.894 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:15:12.117 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:15:12.191 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:15:12.207 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:15:12.207 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:15:12.207 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:15:12.207 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:13.208 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:15:13.209 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:13.209 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:13.209 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:15:13.209 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:13.209 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:13.528 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados-devel 2024-09-22T03:15:14.129 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:28 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:14.262 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:14.262 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:14.262 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-22T03:15:14.262 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:14.263 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:15:14.263 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-22T03:15:14.263 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:14.263 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:15:14.263 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:14.263 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-22T03:15:14.263 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:14.263 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 127 k 2024-09-22T03:15:14.264 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 457 k 2024-09-22T03:15:14.264 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:15:14.477 INFO:teuthology.orchestra.run.smithi070.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 595 kB/s | 127 kB 00:00 2024-09-22T03:15:14.478 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:15:14.478 INFO:teuthology.orchestra.run.smithi070.stdout:Total 591 kB/s | 127 kB 00:00 2024-09-22T03:15:14.478 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:15:14.483 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:15:14.483 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:15:14.547 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:15:14.547 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:15:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:15:14.808 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:15.272 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:15.537 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:15.537 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:15.537 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:15.537 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:15:15.537 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:15.538 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:15.793 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs2 2024-09-22T03:15:16.375 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:30 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:16.467 INFO:teuthology.orchestra.run.smithi070.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:15:16.512 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:16.513 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:15:16.513 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:16.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs-devel 2024-09-22T03:15:17.152 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:31 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:17.282 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:17.283 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-22T03:15:17.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:17.284 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 32 k 2024-09-22T03:15:17.284 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 142 k 2024-09-22T03:15:17.284 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:15:17.431 INFO:teuthology.orchestra.run.smithi070.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 216 kB/s | 32 kB 00:00 2024-09-22T03:15:17.431 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:15:17.431 INFO:teuthology.orchestra.run.smithi070.stdout:Total 214 kB/s | 32 kB 00:00 2024-09-22T03:15:17.431 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:15:17.435 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:15:17.435 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:15:17.452 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:15:17.452 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:15:17.536 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:15:17.644 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:17.997 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:18.228 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:18.228 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:18.228 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:18.228 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:15:18.228 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:18.228 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:18.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados2 2024-09-22T03:15:18.998 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:32 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:19.087 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:15:19.130 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:15:19.131 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:19.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librbd1 2024-09-22T03:15:19.785 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:33 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:19.875 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:15:19.918 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:19.919 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:15:19.919 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:19.989 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rados 2024-09-22T03:15:20.567 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:34 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:20.658 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:15:20.702 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:20.703 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:15:20.703 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:20.775 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rgw 2024-09-22T03:15:21.349 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:35 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:21.439 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:15:21.482 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:21.483 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:15:21.483 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:21.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-cephfs 2024-09-22T03:15:22.128 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:36 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:22.217 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:15:22.260 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:22.334 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:15:22.334 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:22.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rbd 2024-09-22T03:15:22.918 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:36 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:23.007 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:15:23.051 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:23.052 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-09-22T03:15:23.052 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:23.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-fuse 2024-09-22T03:15:23.706 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:37 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:23.840 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:23.840 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 85 k 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 256 k 2024-09-22T03:15:23.841 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:15:24.024 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 467 kB/s | 85 kB 00:00 2024-09-22T03:15:24.025 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:15:24.025 INFO:teuthology.orchestra.run.smithi070.stdout:Total 464 kB/s | 85 kB 00:00 2024-09-22T03:15:24.025 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:15:24.032 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:15:24.032 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:15:24.097 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:15:24.097 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:15:24.247 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:15:24.364 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:24.766 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:25.024 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:25.024 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:25.024 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:25.024 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:15:25.024 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:25.024 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:25.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-mirror 2024-09-22T03:15:25.931 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:39 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:26.071 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 3.1 M 2024-09-22T03:15:26.072 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 14 M 2024-09-22T03:15:26.073 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:15:26.404 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 9.2 MB/s | 3.1 MB 00:00 2024-09-22T03:15:26.405 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:15:26.405 INFO:teuthology.orchestra.run.smithi070.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-22T03:15:26.406 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:15:26.414 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:15:26.414 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:15:26.497 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:15:26.497 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:15:26.938 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:15:26.988 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:27.014 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:27.014 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:15:27.015 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:15:27.015 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:15:27.015 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:15:27.015 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:28.080 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:28.081 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:28.081 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:28.081 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:15:28.081 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:28.081 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:28.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-nbd 2024-09-22T03:15:28.928 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:42 ago on Sun 22 Sep 2024 03:12:46 AM UTC. 2024-09-22T03:15:29.063 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-09-22T03:15:29.063 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:29.063 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-09-22T03:15:29.064 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:29.064 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-09-22T03:15:29.064 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 169 k 2024-09-22T03:15:29.064 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:29.064 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-09-22T03:15:29.064 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-09-22T03:15:29.064 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-09-22T03:15:29.064 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:29.065 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 169 k 2024-09-22T03:15:29.065 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 540 k 2024-09-22T03:15:29.065 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-09-22T03:15:29.283 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 778 kB/s | 169 kB 00:00 2024-09-22T03:15:29.284 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:15:29.284 INFO:teuthology.orchestra.run.smithi070.stdout:Total 771 kB/s | 169 kB 00:00 2024-09-22T03:15:29.284 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-09-22T03:15:29.292 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-09-22T03:15:29.292 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-09-22T03:15:29.357 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-09-22T03:15:29.357 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-09-22T03:15:29.508 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-09-22T03:15:29.614 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:30.048 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:30.323 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:15:30.323 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-09-22T03:15:30.323 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:15:30.323 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:30.323 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-09-22T03:15:30.594 DEBUG:teuthology.parallel:result is None 2024-09-22T03:15:30.594 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:15:30.742 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:15:30.764 INFO:teuthology.orchestra.run.smithi070.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-22T03:15:30.764 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-22T03:15:30.764 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-22T03:15:30.766 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T03:15:30.766 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:30.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:15:30.829 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T03:15:30.829 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:30.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:15:30.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:15:30.957 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T03:15:30.958 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:30.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:15:31.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:15:31.086 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T03:15:31.087 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:31.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:15:31.151 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:15:31.215 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T03:15:31.292 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T03:15:31.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /var/log/ceph 2024-09-22T03:15:31.316 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T03:15:31.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:15:31.380 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T03:15:31.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:15:31.453 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T03:15:31.453 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T03:15:31.454 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914750', 'branch': 'squid', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914750', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'squid', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1158, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi070.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHZop/epIufQV+YLQUqNrG6SwAwvBD9Dz6tGqTUcor19a4MBCJCK0GhmKqlWGPGW22iu81oUCy5kTF/fnj6sGT8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}, 'env': {'RBD_FEATURES': '61'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T03:15:31.454 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:15:31.503 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:15:31.568 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:31.568 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:15:31.624 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:15:31.624 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-09-22T03:15:31.679 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:15:31.680 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:15:31.680 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-22T03:15:31.680 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:15:31.680 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:15:31.680 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-22 03:15:27.245363699 +0000 2024-09-22T03:15:31.680 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-22 03:12:08.979639971 +0000 2024-09-22T03:15:31.680 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-22 03:12:08.979639971 +0000 2024-09-22T03:15:31.680 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-22 03:12:08.979639971 +0000 2024-09-22T03:15:31.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:15:31.742 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-22T03:15:31.742 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-22T03:15:31.742 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.3634e-05 s, 6.1 MB/s 2024-09-22T03:15:31.743 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:15:31.799 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-09-22T03:15:31.854 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:15:31.854 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:15:31.854 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-22T03:15:31.855 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:15:31.855 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:15:31.855 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-22 03:15:27.245363699 +0000 2024-09-22T03:15:31.855 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-22 03:12:09.295635895 +0000 2024-09-22T03:15:31.855 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-22 03:12:09.295635895 +0000 2024-09-22T03:15:31.855 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-22 03:12:09.295635895 +0000 2024-09-22T03:15:31.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-22T03:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-22T03:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 7.7176e-05 s, 6.6 MB/s 2024-09-22T03:15:31.918 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:15:31.974 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-09-22T03:15:32.029 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:15:32.029 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:15:32.029 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-22T03:15:32.030 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:15:32.030 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:15:32.030 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-22 03:15:27.246363686 +0000 2024-09-22T03:15:32.030 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-22 03:12:09.606631883 +0000 2024-09-22T03:15:32.030 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-22 03:12:09.606631883 +0000 2024-09-22T03:15:32.030 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-22 03:12:09.606631883 +0000 2024-09-22T03:15:32.030 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:15:32.092 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-22T03:15:32.092 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-22T03:15:32.092 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.0387e-05 s, 6.4 MB/s 2024-09-22T03:15:32.093 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:15:32.149 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-09-22T03:15:32.204 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:15:32.204 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:15:32.205 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-22T03:15:32.205 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:15:32.205 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:15:32.205 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-22 03:15:27.246363686 +0000 2024-09-22T03:15:32.205 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-22 03:12:09.905628026 +0000 2024-09-22T03:15:32.205 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-22 03:12:09.905628026 +0000 2024-09-22T03:15:32.205 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-09-22 03:12:09.905628026 +0000 2024-09-22T03:15:32.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-22T03:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-22T03:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000104695 s, 4.9 MB/s 2024-09-22T03:15:32.269 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:15:32.325 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:15:32.325 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:15:32.325 INFO:tasks.ceph:Generating config... 2024-09-22T03:15:32.326 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-22T03:15:32.326 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-22T03:15:32.326 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-22T03:15:32.326 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-22T03:15:32.326 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-22T03:15:32.326 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-22T03:15:32.326 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-22T03:15:32.327 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-22T03:15:32.327 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T03:15:32.327 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T03:15:32.327 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T03:15:32.327 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T03:15:32.327 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-22T03:15:32.327 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-22T03:15:32.327 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-22T03:15:32.327 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T03:15:32.327 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:15:32.327 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T03:15:32.327 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T03:15:32.327 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T03:15:32.327 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T03:15:32.327 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T03:15:32.327 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T03:15:32.328 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T03:15:32.328 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T03:15:32.328 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-22T03:15:32.328 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-22T03:15:32.328 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-22T03:15:32.328 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T03:15:32.328 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T03:15:32.444 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T03:15:32.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T03:15:32.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:15:32.519 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.70')] 2024-09-22T03:15:32.519 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.70', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-22T03:15:32.520 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:32.520 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:15:32.575 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.70 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:15:32.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:32.660+0000 7fad5549fe80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:32.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:32.660+0000 7fad5549fe80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:32.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:32.660+0000 7fad5549fe80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:32.663 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:15:32.663 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: generated fsid f1076ac7-a939-441d-8eeb-e35612888a01 2024-09-22T03:15:32.663 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = quincy 2024-09-22T03:15:32.663 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 0 2024-09-22T03:15:32.663 INFO:teuthology.orchestra.run.smithi070.stdout:fsid f1076ac7-a939-441d-8eeb-e35612888a01 2024-09-22T03:15:32.663 INFO:teuthology.orchestra.run.smithi070.stdout:last_changed 2024-09-22T03:15:32.661933+0000 2024-09-22T03:15:32.664 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-09-22T03:15:32.661933+0000 2024-09-22T03:15:32.664 INFO:teuthology.orchestra.run.smithi070.stdout:min_mon_release 17 (quincy) 2024-09-22T03:15:32.664 INFO:teuthology.orchestra.run.smithi070.stdout:election_strategy: 1 2024-09-22T03:15:32.664 INFO:teuthology.orchestra.run.smithi070.stdout:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-09-22T03:15:32.664 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T03:15:32.665 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:15:32.680 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f1076ac7-a939-441d-8eeb-e35612888a01... 2024-09-22T03:15:32.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:15:32.763 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-09-22T03:15:32.763 INFO:teuthology.orchestra.run.smithi070.stdout: chdir = "" 2024-09-22T03:15:32.763 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:15:32.763 INFO:teuthology.orchestra.run.smithi070.stdout: auth supported = cephx 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: filestore xattr use omap = true 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: mon clock drift allowed = 1.000 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: osd crush chooseleaf type = 0 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: auth debug = true 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on old message = true 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on bug = true 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: mon pg warn max object skew = 0 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.764 INFO:teuthology.orchestra.run.smithi070.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default size = 2 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow primary affinity = true 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow pg remap = true 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on no sortbitwise = false 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on too few osds = false 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: mon_allow_pool_size_one = true 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: osd default data pool replay window = 5 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: mon allow pool delete = true 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: mon cluster log file level = debug 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: debug asserts on shutdown = true 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: mon health detail to clog = false 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: mon host = 172.21.15.70 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: ms client mode = crc 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: ms cluster mode = crc 2024-09-22T03:15:32.766 INFO:teuthology.orchestra.run.smithi070.stdout: ms mon client mode = crc 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: ms mon cluster mode = crc 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: ms mon service mode = crc 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: ms service mode = crc 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = f1076ac7-a939-441d-8eeb-e35612888a01 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout:[osd] 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: osd journal size = 100 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub load threshold = 5.0 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub max interval = 600 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: osd recover clone overlap = true 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: osd recovery max chunk = 1048576 2024-09-22T03:15:32.767 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug shutdown = true 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug op order = true 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify stray on activate = true 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug trim objects = true 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: osd open classes on start = true 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug pg log writeout = true 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: osd map max advance = 10 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: journal zero on create = true 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:15:32.768 INFO:teuthology.orchestra.run.smithi070.stdout: filestore apply finisher threads = 3 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: bdev debug aio = true 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug misdirected ops = true 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: debug monc = 20 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: crimson osd obc lru size = 10 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: debug osd = 20 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: osd objectstore = seastore 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: osd shutdown pgref assert = True 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr] 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: debug mgr = 20 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-09-22T03:15:32.769 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mgr/telemetry/nag = false 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout:[mon] 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: debug paxos = 20 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mon data avail warn = 5 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mon mgr mkfs grace = 240 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd reporter subtree level = osd 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd prime pg temp = true 2024-09-22T03:15:32.770 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: # 1m isn't quite enough 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:15:32.771 INFO:teuthology.orchestra.run.smithi070.stdout: mon min osdmap epochs = 50 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune interval = 2 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune min = 15 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune txsize = 2 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: paxos service trim min = 10 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default crimson = True 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout:[client] 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: rgw cache enabled = true 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable ops log = true 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable usage log = true 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:15:32.772 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.a] 2024-09-22T03:15:32.773 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-09-22T03:15:32.773 INFO:teuthology.orchestra.run.smithi070.stdout: crimson seastar cpu cores = 0-1 2024-09-22T03:15:32.773 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-09-22T03:15:32.773 INFO:teuthology.orchestra.run.smithi070.stdout: crimson seastar cpu cores = 2-3 2024-09-22T03:15:32.773 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-09-22T03:15:32.773 INFO:teuthology.orchestra.run.smithi070.stdout: crimson seastar cpu cores = 4-5 2024-09-22T03:15:32.774 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T03:15:32.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T03:15:32.874 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T03:15:32.874 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:32.875 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T03:15:32.890 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:32.890 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T03:15:32.946 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:15:32.946 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:32.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:15:32.946 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:15:33.020 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:33.020 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:15:33.075 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T03:15:33.075 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T03:15:33.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:15:33.171 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:15:33.173 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T03:15:33.173 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T03:15:33.174 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T03:15:33.220 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T03:15:33.232 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T03:15:33.233 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:15:33.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T03:15:33.257 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:15:33.257 INFO:tasks.ceph:role: osd.0 2024-09-22T03:15:33.257 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:15:33.257 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:15:33.322 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:15:33.354 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-22T03:15:33.356 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-22T03:15:33.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T03:15:33.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T03:15:33.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T03:15:33.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:15:33.562 INFO:tasks.ceph:role: osd.1 2024-09-22T03:15:33.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:15:33.562 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:15:33.626 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:15:33.626 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:15:33.626 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:15:33.626 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:15:33.626 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:15:33.626 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:15:33.626 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:15:33.626 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:15:33.627 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:15:33.627 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:15:33.659 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-22T03:15:33.661 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-22T03:15:33.661 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T03:15:33.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T03:15:33.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T03:15:33.865 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:15:33.866 INFO:tasks.ceph:role: osd.2 2024-09-22T03:15:33.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:15:33.866 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-22T03:15:33.929 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:15:33.929 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:15:33.929 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:15:33.930 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:15:33.930 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:15:33.930 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:15:33.930 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:15:33.930 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:15:33.930 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:15:33.930 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:15:33.962 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-22T03:15:33.964 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-22T03:15:33.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-22T03:15:34.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T03:15:34.101 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:15:34.647 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:15:34.648 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:34,647 seastar - Reactor backend: linux-aio 2024-09-22T03:15:34.666 INFO:teuthology.orchestra.run.smithi070.stderr:==27675==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:15:34.710 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:34,709 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:15:34.710 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:34,709 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-22T03:15:34.710 INFO:teuthology.orchestra.run.smithi070.stderr:DEBUG 2024-09-22 03:15:34,710 [shard 0:main] seastar - reactor::drain 2024-09-22T03:15:35.191 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:35.191 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-09-22T03:15:35.191 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-09-22T03:15:35.191 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-09-22T03:15:35.191 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:35.191 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-22T03:15:35.246 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:15:35.248 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:35,247 seastar - Reactor backend: linux-aio 2024-09-22T03:15:35.276 INFO:teuthology.orchestra.run.smithi070.stderr:==27673==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:15:35.427 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2ff75b41 0x2ff7779a 0x3014ba39 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:35.427 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:35.460 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x30151414 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:35.460 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:35.512 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c5f1 0x2857c 0xb6323 0x30149b1b 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:35.512 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffaa08c7a4 0xffffffffaacd1db2 0xffffffffaae00bb2 2024-09-22T03:15:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b79 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:35.592 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b2d 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:35.592 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:35.728 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x20520951 0x2054f521 0x205d4515 0x30071ab6 0x3007230a 0x3007d9e0 0x30081d6a 0x2b8ab682 0x2bd807cb 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:35.728 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:35.785 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095a19 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:35.786 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:35.815 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b15 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:35.815 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:36.433 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:36.434 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-09-22T03:15:36.434 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-09-22T03:15:36.434 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-09-22T03:15:36.434 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:36.434 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-22T03:15:36.434 INFO:teuthology.orchestra.run.smithi070.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid f1076ac7-a939-441d-8eeb-e35612888a01 2024-09-22T03:15:36.476 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T03:15:36.500 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:15:37.041 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:15:37.042 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:37,041 seastar - Reactor backend: linux-aio 2024-09-22T03:15:37.060 INFO:teuthology.orchestra.run.smithi070.stderr:==27720==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:15:37.105 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:37,104 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:15:37.105 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:37,104 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-22T03:15:37.105 INFO:teuthology.orchestra.run.smithi070.stderr:DEBUG 2024-09-22 03:15:37,104 [shard 0:main] seastar - reactor::drain 2024-09-22T03:15:37.580 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:37.580 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-09-22T03:15:37.580 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-09-22T03:15:37.580 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-09-22T03:15:37.580 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:37.580 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-22T03:15:37.639 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:15:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:37,640 seastar - Reactor backend: linux-aio 2024-09-22T03:15:37.699 INFO:teuthology.orchestra.run.smithi070.stderr:==27718==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:15:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x8b581 0xc384c 0x2c57d 0x2857c 0xb6323 0x2fcc5109 0x2fcc5a6e 0x2fd9c974 0x2fd9d677 0x2bd80969 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:37.846 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x161f40 0x94469 0x94d46 0x2ffb1c03 0x2ffc4e07 0x3014e2a4 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:37.846 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:37.898 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xb634f 0x2fcc5109 0x2fcc5a6e 0x2fd9c974 0x2fd9d677 0x2bd80969 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:37.898 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:37.925 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095984 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:37.925 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095acb 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:38.110 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xcdfc3 0xd208d 0x2bc18 0x2c061 0x2c0f0 0x2c3b8 0x2c5a7 0x2857c 0xb6323 0x2fcc7347 0x3015162d 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:38.110 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffaa3f7f30 0xffffffffaa6904d5 0xffffffffaa694f34 0xffffffffaa39c3f8 0xffffffffaa39df72 0xffffffffaa39e118 0xffffffffaa39e522 0xffffffffaa39fe99 0xffffffffaa3a0afd 0xffffffffaa36ccad 0xffffffffaaccd45f 0xffffffffaae00130 2024-09-22T03:15:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c0959ea 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b15 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-09-22T03:15:38.820 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-09-22T03:15:38.820 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-09-22T03:15:38.820 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:38.820 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-22T03:15:38.820 INFO:teuthology.orchestra.run.smithi070.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid f1076ac7-a939-441d-8eeb-e35612888a01 2024-09-22T03:15:38.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T03:15:38.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:15:39.433 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:15:39.434 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:39,433 seastar - Reactor backend: linux-aio 2024-09-22T03:15:39.452 INFO:teuthology.orchestra.run.smithi070.stderr:==27765==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:15:39.495 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:39,494 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-22T03:15:39.495 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:39,494 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-22T03:15:39.495 INFO:teuthology.orchestra.run.smithi070.stderr:DEBUG 2024-09-22 03:15:39,495 [shard 0:main] seastar - reactor::drain 2024-09-22T03:15:39.976 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:39.976 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-09-22T03:15:39.976 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-09-22T03:15:39.976 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-09-22T03:15:39.976 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:39.976 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-22T03:15:40.031 INFO:teuthology.orchestra.run.smithi070.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:15:40.033 INFO:teuthology.orchestra.run.smithi070.stderr:INFO 2024-09-22 03:15:40,032 seastar - Reactor backend: linux-aio 2024-09-22T03:15:40.052 INFO:teuthology.orchestra.run.smithi070.stderr:==27763==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:15:40.202 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xd5958 0xd5ac0 0xd554b 0x2c64a 0x282da 0xb49e1 0x2077e3b7 0x2fcc56c8 0x2fd9c974 0x2fd9d677 0x2bd80969 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:40.202 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:40.235 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c6bb 0x282da 0xb49e1 0x2fd9afe6 0x2fd9d677 0x2bd7f080 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:40.235 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:40.286 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x3007dcbf 0x2b8ab682 0x2bd7eef6 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2b28912b 0x2b3a7096 0x2b532008 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:40.286 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:40.314 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095adc 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:40.314 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:40.363 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095a80 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:40.363 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:40.497 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c605 0x2857c 0xb6323 0x30040555 0x30041e94 0x3004afd5 0x3014aa5f 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:40.497 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffaa41e6af 0xffffffffaa41f2c2 0xffffffffaa420b49 0xffffffffaa38e01c 0xffffffffaa39699b 0xffffffffaa396e1c 0xffffffffaa08c884 0xffffffffaacd1db2 0xffffffffaae00bb2 2024-09-22T03:15:40.554 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095aaf 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:15:40.554 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:40.582 INFO:teuthology.orchestra.run.smithi070.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095b44 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:15:40.582 INFO:teuthology.orchestra.run.smithi070.stderr:kernel callstack: 2024-09-22T03:15:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:Suppressions used: 2024-09-22T03:15:41.191 INFO:teuthology.orchestra.run.smithi070.stderr: count bytes template 2024-09-22T03:15:41.191 INFO:teuthology.orchestra.run.smithi070.stderr: 1 8 MallocExtension::Register 2024-09-22T03:15:41.191 INFO:teuthology.orchestra.run.smithi070.stderr:----------------------------------------------------- 2024-09-22T03:15:41.191 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-22T03:15:41.192 INFO:teuthology.orchestra.run.smithi070.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid f1076ac7-a939-441d-8eeb-e35612888a01 2024-09-22T03:15:41.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T03:15:41.260 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T03:15:41.260 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:41.260 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T03:15:41.325 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:41.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T03:15:41.389 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:41.389 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T03:15:41.452 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:41.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T03:15:41.517 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:41.517 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T03:15:41.574 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T03:15:41.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.x] 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDVi+9mXI0kChAAhTqvipW3qTwfQb3+vvzg7w== 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDXi+9mlFHDExAA3N8QQ23xNPrasKtOF8OGAg== 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDZi+9mtLSpKhAA7qPtUIF/BnJMkBuWMSgq+A== 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDci+9mR4JaBhAAr9D9KJVzPAQA4XxplPAMOw== 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-09-22T03:15:41.637 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDVi+9mKFEUDRAAkmWvrEk3XB9YgPn9X4qyYA== 2024-09-22T03:15:41.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T03:15:41.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:15:41.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:15:41.848 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:15:41.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:15:41.964 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T03:15:41.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T03:15:41.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T03:15:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:42.095+0000 7f482528eb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:42.095+0000 7f482528eb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T03:15:42.134 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:15:42.189 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T03:15:42.189 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T03:15:42.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T03:15:42.232 INFO:tasks.ceph.mon.a:Started 2024-09-22T03:15:42.232 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T03:15:42.232 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T03:15:42.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T03:15:42.235 INFO:tasks.ceph.mgr.x:Started 2024-09-22T03:15:42.235 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T03:15:42.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T03:15:42.311 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-09-22T03:15:42.309+0000 7f51de88cb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.311 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-09-22T03:15:42.309+0000 7f51de88cb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.311 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-09-22T03:15:42.309+0000 7f51de88cb40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.312 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:42.310+0000 7f761bbbdf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:42.337+0000 7fb89f797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:42.346+0000 7fb89f797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.414 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-09-22T03:15:42.426 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T03:15:42.426 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T03:15:42.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T03:15:42.516 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:42.514+0000 7f761bbbdf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.516 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:42.515+0000 7f761bbbdf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:42.521+0000 7f457bd97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:42.528+0000 7f457bd97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:42.608 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-09-22T03:15:42.606+0000 7f51d5bf1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T03:15:42.608 INFO:tasks.ceph.mon.a.smithi070.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T03:15:42.608 INFO:tasks.ceph.mon.a.smithi070.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T03:15:42.608 INFO:tasks.ceph.mon.a.smithi070.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T03:15:42.608 INFO:tasks.ceph.mon.a.smithi070.stderr:feature with important data. 2024-09-22T03:15:42.609 INFO:tasks.ceph.mon.a.smithi070.stderr: 2024-09-22T03:15:42.617 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:42.615+0000 7f761bbbdf00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-22T03:15:42.657 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:42.655+0000 7f761bbbdf00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-22T03:15:42.784 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:42.782+0000 7f761bbbdf00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-22T03:15:42.833 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:42.831+0000 7f761bbbdf00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-22T03:15:43.008 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:43.006+0000 7f761bbbdf00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:15:43.117 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:43.116+0000 7f761bbbdf00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-22T03:15:43.426 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T03:15:43.426 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:43.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T03:15:43.434 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:43.432+0000 7f761bbbdf00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-22T03:15:43.451 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:43.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T03:15:43.491 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:43.489+0000 7f761bbbdf00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-22T03:15:43.515 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:15:43.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T03:15:43.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new adeb97d5-5208-4d0d-b820-010b67e342a7 0 2024-09-22T03:15:43.660 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:43.659+0000 7f761bbbdf00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-22T03:15:43.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:43.722+0000 7fb5cc197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:43.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:43.731+0000 7fb5cc197640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:43.808 INFO:teuthology.orchestra.run.smithi070.stdout:0 2024-09-22T03:15:43.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new 8f02f123-7480-4f9e-ab4a-b72de52061d9 1 2024-09-22T03:15:43.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:43.955+0000 7f88c7858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:43.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:43.963+0000 7f88c7858640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:44.041 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-09-22T03:15:44.053 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new b4c0fde8-059a-4132-b1e7-872352bb8aa7 2 2024-09-22T03:15:44.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:44.148+0000 7f5b41997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:44.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:44.157+0000 7f5b41997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:44.249 INFO:teuthology.orchestra.run.smithi070.stdout:2 2024-09-22T03:15:44.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T03:15:44.290 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:15:44.290 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:15:44.333 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:15:44.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T03:15:44.362 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:15:44.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:15:44.404 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:15:44.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T03:15:44.435 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:15:44.435 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:15:44.478 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:15:44.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:15:44.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:44.665+0000 7fc062aba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:44.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:44.678+0000 7fc062aba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:44.773 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:44.771+0000 7f761bbbdf00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-22T03:15:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:44.814 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":6,"fsid":"f1076ac7-a939-441d-8eeb-e35612888a01","created":"2024-09-22T03:15:42.327011+0000","modified":"2024-09-22T03:15:44.247845+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:15:44.247845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"adeb97d5-5208-4d0d-b820-010b67e342a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f02f123-7480-4f9e-ab4a-b72de52061d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b4c0fde8-059a-4132-b1e7-872352bb8aa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:15:44.836 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T03:15:44.837 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T03:15:44.838 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:44.837+0000 7f761bbbdf00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-22T03:15:45.139 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:15:45.235 INFO:teuthology.misc.health.smithi070.stderr:2024-09-22T03:15:45.230+0000 7f5d2664f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:45.238 INFO:teuthology.misc.health.smithi070.stderr:2024-09-22T03:15:45.236+0000 7f5d2664f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:45.303 INFO:teuthology.misc.health.smithi070.stdout: 2024-09-22T03:15:45.303 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":6,"fsid":"f1076ac7-a939-441d-8eeb-e35612888a01","created":"2024-09-22T03:15:42.327011+0000","modified":"2024-09-22T03:15:44.247845+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:15:44.247845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"adeb97d5-5208-4d0d-b820-010b67e342a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f02f123-7480-4f9e-ab4a-b72de52061d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b4c0fde8-059a-4132-b1e7-872352bb8aa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:15:45.313 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-22T03:15:45.359 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:45.357+0000 7f761bbbdf00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-22T03:15:45.470 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:45.468+0000 7f761bbbdf00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-22T03:15:45.656 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:45.654+0000 7f761bbbdf00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:15:45.732 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:45.731+0000 7f761bbbdf00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-22T03:15:45.923 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:45.921+0000 7f761bbbdf00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-22T03:15:46.006 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:46.004+0000 7f761bbbdf00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-22T03:15:46.077 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:46.076+0000 7f761bbbdf00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-22T03:15:46.266 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:46.264+0000 7f761bbbdf00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-22T03:15:46.346 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:46.344+0000 7f761bbbdf00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-22T03:15:46.429 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:46.427+0000 7f761bbbdf00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-22T03:15:46.541 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:46.539+0000 7f761bbbdf00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-22T03:15:46.613 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:46.611+0000 7f761bbbdf00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-22T03:15:46.683 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:46.682+0000 7f761bbbdf00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-22T03:15:47.381 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:47.379+0000 7f761bbbdf00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-22T03:15:48.390 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-09-22T03:15:48.388+0000 7f75ba4ac640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T03:15:51.616 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:15:51.721 INFO:teuthology.misc.health.smithi070.stderr:2024-09-22T03:15:51.716+0000 7fe688c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:51.724 INFO:teuthology.misc.health.smithi070.stderr:2024-09-22T03:15:51.723+0000 7fe688c63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:51.901 INFO:teuthology.misc.health.smithi070.stdout: 2024-09-22T03:15:51.901 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":9,"fsid":"f1076ac7-a939-441d-8eeb-e35612888a01","created":"2024-09-22T03:15:42.327011+0000","modified":"2024-09-22T03:15:51.441048+0000","last_up_change":"2024-09-22T03:15:49.436070+0000","last_in_change":"2024-09-22T03:15:44.247845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:15:51.416295+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"adeb97d5-5208-4d0d-b820-010b67e342a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3847971085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":3847971085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3847971085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3847971085}]},"public_addr":"172.21.15.70:6804/3847971085","cluster_addr":"172.21.15.70:6805/3847971085","heartbeat_back_addr":"172.21.15.70:6813/3847971085","heartbeat_front_addr":"172.21.15.70:6812/3847971085","state":["exists","up"]},{"osd":1,"uuid":"8f02f123-7480-4f9e-ab4a-b72de52061d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":3697286852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697286852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":3697286852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697286852}]},"public_addr":"172.21.15.70:6803/3697286852","cluster_addr":"172.21.15.70:6802/3697286852","heartbeat_back_addr":"172.21.15.70:6807/3697286852","heartbeat_front_addr":"172.21.15.70:6806/3697286852","state":["exists","up"]},{"osd":2,"uuid":"b4c0fde8-059a-4132-b1e7-872352bb8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":782143372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":782143372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6815","nonce":782143372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":782143372}]},"public_addr":"172.21.15.70:6810/782143372","cluster_addr":"172.21.15.70:6809/782143372","heartbeat_back_addr":"172.21.15.70:6815/782143372","heartbeat_front_addr":"172.21.15.70:6814/782143372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:15:51.912 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-22T03:15:51.912 INFO:tasks.ceph:Creating RBD pool 2024-09-22T03:15:51.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T03:15:52.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:52.005+0000 7ff633597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:52.015+0000 7ff633597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.446 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'rbd' created 2024-09-22T03:15:52.465 DEBUG:teuthology.orchestra.run.smithi070:> rbd --cluster ceph pool init rbd 2024-09-22T03:15:52.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:52.519+0000 7f9d3b841d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:52.519+0000 7f9d3b841d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:52.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:52.524+0000 7f9d3b841d00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:55.491 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T03:15:55.492 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T03:15:55.492 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T03:15:55.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T03:15:55.492 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T03:15:55.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:55.593+0000 7fb2dba6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:55.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:55.599+0000 7fb2dba6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:55.822 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:55.834 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2812640414},{"type":"v1","addr":"172.21.15.70:6801","nonce":2812640414}]},"active_addr":"172.21.15.70:6801/2812640414","active_change":"2024-09-22T03:15:47.381693+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":4007431615}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3250977213}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":1978029621}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":4162832386}]}]} 2024-09-22T03:15:55.836 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T03:15:55.836 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:15:55.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:15:55.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:55.933+0000 7f3e8f263640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:55.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:55.940+0000 7f3e8f263640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.117 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:56.117 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":13,"fsid":"f1076ac7-a939-441d-8eeb-e35612888a01","created":"2024-09-22T03:15:42.327011+0000","modified":"2024-09-22T03:15:55.454586+0000","last_up_change":"2024-09-22T03:15:49.436070+0000","last_in_change":"2024-09-22T03:15:44.247845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:15:51.416295+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:15:52.207511+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"adeb97d5-5208-4d0d-b820-010b67e342a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3847971085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":3847971085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3847971085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3847971085}]},"public_addr":"172.21.15.70:6804/3847971085","cluster_addr":"172.21.15.70:6805/3847971085","heartbeat_back_addr":"172.21.15.70:6813/3847971085","heartbeat_front_addr":"172.21.15.70:6812/3847971085","state":["exists","up"]},{"osd":1,"uuid":"8f02f123-7480-4f9e-ab4a-b72de52061d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":3697286852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697286852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":3697286852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697286852}]},"public_addr":"172.21.15.70:6803/3697286852","cluster_addr":"172.21.15.70:6802/3697286852","heartbeat_back_addr":"172.21.15.70:6807/3697286852","heartbeat_front_addr":"172.21.15.70:6806/3697286852","state":["exists","up"]},{"osd":2,"uuid":"b4c0fde8-059a-4132-b1e7-872352bb8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":782143372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":782143372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6815","nonce":782143372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":782143372}]},"public_addr":"172.21.15.70:6810/782143372","cluster_addr":"172.21.15.70:6809/782143372","heartbeat_back_addr":"172.21.15.70:6815/782143372","heartbeat_front_addr":"172.21.15.70:6814/782143372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:15:56.129 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:15:56.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:15:56.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.226+0000 7f21a3d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.232+0000 7f21a3d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.411 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:56.412 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":13,"fsid":"f1076ac7-a939-441d-8eeb-e35612888a01","created":"2024-09-22T03:15:42.327011+0000","modified":"2024-09-22T03:15:55.454586+0000","last_up_change":"2024-09-22T03:15:49.436070+0000","last_in_change":"2024-09-22T03:15:44.247845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:15:51.416295+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:15:52.207511+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"adeb97d5-5208-4d0d-b820-010b67e342a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3847971085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":3847971085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3847971085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3847971085}]},"public_addr":"172.21.15.70:6804/3847971085","cluster_addr":"172.21.15.70:6805/3847971085","heartbeat_back_addr":"172.21.15.70:6813/3847971085","heartbeat_front_addr":"172.21.15.70:6812/3847971085","state":["exists","up"]},{"osd":1,"uuid":"8f02f123-7480-4f9e-ab4a-b72de52061d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":3697286852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697286852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":3697286852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697286852}]},"public_addr":"172.21.15.70:6803/3697286852","cluster_addr":"172.21.15.70:6802/3697286852","heartbeat_back_addr":"172.21.15.70:6807/3697286852","heartbeat_front_addr":"172.21.15.70:6806/3697286852","state":["exists","up"]},{"osd":2,"uuid":"b4c0fde8-059a-4132-b1e7-872352bb8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":782143372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":782143372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6815","nonce":782143372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":782143372}]},"public_addr":"172.21.15.70:6810/782143372","cluster_addr":"172.21.15.70:6809/782143372","heartbeat_back_addr":"172.21.15.70:6815/782143372","heartbeat_front_addr":"172.21.15.70:6814/782143372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:15:56.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:15:56.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:15:56.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:15:56.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.527+0000 7f6d19ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.527+0000 7f577f797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.528+0000 7f76f9997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.535+0000 7f577f797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.535+0000 7f6d19ab2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.536+0000 7f76f9997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.565 INFO:teuthology.orchestra.run.smithi070.stdout:34359738375 2024-09-22T03:15:56.566 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:15:56.566 INFO:teuthology.orchestra.run.smithi070.stdout:34359738375 2024-09-22T03:15:56.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:15:56.567 INFO:teuthology.orchestra.run.smithi070.stdout:34359738375 2024-09-22T03:15:56.568 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:15:56.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.667+0000 7f3486157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.667+0000 7f582f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.668+0000 7f56bed97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.675+0000 7f3486157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.675+0000 7f582f997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:56.676+0000 7f56bed97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:56.871 INFO:teuthology.orchestra.run.smithi070.stdout:34359738375 2024-09-22T03:15:56.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2024-09-22T03:15:56.884 DEBUG:teuthology.parallel:result is None 2024-09-22T03:15:56.885 INFO:teuthology.orchestra.run.smithi070.stdout:34359738375 2024-09-22T03:15:56.893 INFO:teuthology.orchestra.run.smithi070.stdout:34359738372 2024-09-22T03:15:56.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.0 2024-09-22T03:15:56.898 DEBUG:teuthology.parallel:result is None 2024-09-22T03:15:56.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738372 for osd.2 2024-09-22T03:15:57.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:15:58.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:58.017+0000 7fe8660d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:58.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:58.025+0000 7fe8660d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:58.203 INFO:teuthology.orchestra.run.smithi070.stdout:34359738375 2024-09-22T03:15:58.216 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.2 2024-09-22T03:15:58.217 DEBUG:teuthology.parallel:result is None 2024-09-22T03:15:58.217 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:15:58.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:15:58.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:58.313+0000 7fc7e8d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:58.319+0000 7fc7e8d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:58.498 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:58.499 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-22T03:15:58.510 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T03:15:57.386098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3324,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889860,"statfs":{"total":160658620416,"available":160655216640,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.834165"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.470298+0000","last_change":"2024-09-22T03:15:55.475462+0000","last_active":"2024-09-22T03:15:55.470298+0000","last_peered":"2024-09-22T03:15:55.470298+0000","last_clean":"2024-09-22T03:15:55.470298+0000","last_became_active":"2024-09-22T03:15:53.559125+0000","last_became_peered":"2024-09-22T03:15:53.559125+0000","last_unstale":"2024-09-22T03:15:55.470298+0000","last_undegraded":"2024-09-22T03:15:55.470298+0000","last_fullsized":"2024-09-22T03:15:55.470298+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.472651+0000","last_change":"2024-09-22T03:15:55.476029+0000","last_active":"2024-09-22T03:15:55.472651+0000","last_peered":"2024-09-22T03:15:55.472651+0000","last_clean":"2024-09-22T03:15:55.472651+0000","last_became_active":"2024-09-22T03:15:53.591886+0000","last_became_peered":"2024-09-22T03:15:53.591886+0000","last_unstale":"2024-09-22T03:15:55.472651+0000","last_undegraded":"2024-09-22T03:15:55.472651+0000","last_fullsized":"2024-09-22T03:15:55.472651+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.470731+0000","last_change":"2024-09-22T03:15:55.475380+0000","last_active":"2024-09-22T03:15:55.470731+0000","last_peered":"2024-09-22T03:15:55.470731+0000","last_clean":"2024-09-22T03:15:55.470731+0000","last_became_active":"2024-09-22T03:15:53.569486+0000","last_became_peered":"2024-09-22T03:15:53.569486+0000","last_unstale":"2024-09-22T03:15:55.470731+0000","last_undegraded":"2024-09-22T03:15:55.470731+0000","last_fullsized":"2024-09-22T03:15:55.470731+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.473137+0000","last_change":"2024-09-22T03:15:55.475811+0000","last_active":"2024-09-22T03:15:55.473137+0000","last_peered":"2024-09-22T03:15:55.473137+0000","last_clean":"2024-09-22T03:15:55.473137+0000","last_became_active":"2024-09-22T03:15:53.588564+0000","last_became_peered":"2024-09-22T03:15:53.588564+0000","last_unstale":"2024-09-22T03:15:55.473137+0000","last_undegraded":"2024-09-22T03:15:55.473137+0000","last_fullsized":"2024-09-22T03:15:55.473137+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"12'1","reported_seq":17,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.474905+0000","last_change":"2024-09-22T03:15:55.474905+0000","last_active":"2024-09-22T03:15:55.474905+0000","last_peered":"2024-09-22T03:15:55.474905+0000","last_clean":"2024-09-22T03:15:55.474905+0000","last_became_active":"2024-09-22T03:15:53.579172+0000","last_became_peered":"2024-09-22T03:15:53.579172+0000","last_unstale":"2024-09-22T03:15:55.474905+0000","last_undegraded":"2024-09-22T03:15:55.474905+0000","last_fullsized":"2024-09-22T03:15:55.474905+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.474340+0000","last_change":"2024-09-22T03:15:55.476632+0000","last_active":"2024-09-22T03:15:55.474340+0000","last_peered":"2024-09-22T03:15:55.474340+0000","last_clean":"2024-09-22T03:15:55.474340+0000","last_became_active":"2024-09-22T03:15:53.583253+0000","last_became_peered":"2024-09-22T03:15:53.583253+0000","last_unstale":"2024-09-22T03:15:55.474340+0000","last_undegraded":"2024-09-22T03:15:55.474340+0000","last_fullsized":"2024-09-22T03:15:55.474340+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.470640+0000","last_change":"2024-09-22T03:15:55.473205+0000","last_active":"2024-09-22T03:15:55.470640+0000","last_peered":"2024-09-22T03:15:55.470640+0000","last_clean":"2024-09-22T03:15:55.470640+0000","last_became_active":"2024-09-22T03:15:53.577075+0000","last_became_peered":"2024-09-22T03:15:53.577075+0000","last_unstale":"2024-09-22T03:15:55.470640+0000","last_undegraded":"2024-09-22T03:15:55.470640+0000","last_fullsized":"2024-09-22T03:15:55.470640+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":16,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.468883+0000","last_change":"2024-09-22T03:15:55.474940+0000","last_active":"2024-09-22T03:15:55.468883+0000","last_peered":"2024-09-22T03:15:55.468883+0000","last_clean":"2024-09-22T03:15:55.468883+0000","last_became_active":"2024-09-22T03:15:53.561718+0000","last_became_peered":"2024-09-22T03:15:53.561718+0000","last_unstale":"2024-09-22T03:15:55.468883+0000","last_undegraded":"2024-09-22T03:15:55.468883+0000","last_fullsized":"2024-09-22T03:15:55.468883+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.469731+0000","last_change":"2024-09-22T03:15:52.494206+0000","last_active":"2024-09-22T03:15:55.469731+0000","last_peered":"2024-09-22T03:15:55.469731+0000","last_clean":"2024-09-22T03:15:55.469731+0000","last_became_active":"2024-09-22T03:15:52.482270+0000","last_became_peered":"2024-09-22T03:15:52.482270+0000","last_unstale":"2024-09-22T03:15:55.469731+0000","last_undegraded":"2024-09-22T03:15:55.469731+0000","last_fullsized":"2024-09-22T03:15:55.469731+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_clean_scrub_stamp":"2024-09-22T03:15:51.441048+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655216640,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1580,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296148,"statfs":{"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":420,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297308,"statfs":{"total":53552873472,"available":53552443392,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1324,"kb_used_data":1324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296404,"statfs":{"total":53552873472,"available":53551517696,"internally_reserved":0,"allocated":1355776,"data_stored":1355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551517696,"internally_reserved":0,"allocated":1355776,"data_stored":1355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552443392,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:15:58.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:15:58.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:58.608+0000 7f7938263640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:58.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:58.615+0000 7f7938263640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:58.794 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:58.795 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-22T03:15:58.804 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-22T03:15:57.386098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3324,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889860,"statfs":{"total":160658620416,"available":160655216640,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.834165"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.470298+0000","last_change":"2024-09-22T03:15:55.475462+0000","last_active":"2024-09-22T03:15:55.470298+0000","last_peered":"2024-09-22T03:15:55.470298+0000","last_clean":"2024-09-22T03:15:55.470298+0000","last_became_active":"2024-09-22T03:15:53.559125+0000","last_became_peered":"2024-09-22T03:15:53.559125+0000","last_unstale":"2024-09-22T03:15:55.470298+0000","last_undegraded":"2024-09-22T03:15:55.470298+0000","last_fullsized":"2024-09-22T03:15:55.470298+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.472651+0000","last_change":"2024-09-22T03:15:55.476029+0000","last_active":"2024-09-22T03:15:55.472651+0000","last_peered":"2024-09-22T03:15:55.472651+0000","last_clean":"2024-09-22T03:15:55.472651+0000","last_became_active":"2024-09-22T03:15:53.591886+0000","last_became_peered":"2024-09-22T03:15:53.591886+0000","last_unstale":"2024-09-22T03:15:55.472651+0000","last_undegraded":"2024-09-22T03:15:55.472651+0000","last_fullsized":"2024-09-22T03:15:55.472651+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.470731+0000","last_change":"2024-09-22T03:15:55.475380+0000","last_active":"2024-09-22T03:15:55.470731+0000","last_peered":"2024-09-22T03:15:55.470731+0000","last_clean":"2024-09-22T03:15:55.470731+0000","last_became_active":"2024-09-22T03:15:53.569486+0000","last_became_peered":"2024-09-22T03:15:53.569486+0000","last_unstale":"2024-09-22T03:15:55.470731+0000","last_undegraded":"2024-09-22T03:15:55.470731+0000","last_fullsized":"2024-09-22T03:15:55.470731+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.473137+0000","last_change":"2024-09-22T03:15:55.475811+0000","last_active":"2024-09-22T03:15:55.473137+0000","last_peered":"2024-09-22T03:15:55.473137+0000","last_clean":"2024-09-22T03:15:55.473137+0000","last_became_active":"2024-09-22T03:15:53.588564+0000","last_became_peered":"2024-09-22T03:15:53.588564+0000","last_unstale":"2024-09-22T03:15:55.473137+0000","last_undegraded":"2024-09-22T03:15:55.473137+0000","last_fullsized":"2024-09-22T03:15:55.473137+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"12'1","reported_seq":17,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.474905+0000","last_change":"2024-09-22T03:15:55.474905+0000","last_active":"2024-09-22T03:15:55.474905+0000","last_peered":"2024-09-22T03:15:55.474905+0000","last_clean":"2024-09-22T03:15:55.474905+0000","last_became_active":"2024-09-22T03:15:53.579172+0000","last_became_peered":"2024-09-22T03:15:53.579172+0000","last_unstale":"2024-09-22T03:15:55.474905+0000","last_undegraded":"2024-09-22T03:15:55.474905+0000","last_fullsized":"2024-09-22T03:15:55.474905+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.474340+0000","last_change":"2024-09-22T03:15:55.476632+0000","last_active":"2024-09-22T03:15:55.474340+0000","last_peered":"2024-09-22T03:15:55.474340+0000","last_clean":"2024-09-22T03:15:55.474340+0000","last_became_active":"2024-09-22T03:15:53.583253+0000","last_became_peered":"2024-09-22T03:15:53.583253+0000","last_unstale":"2024-09-22T03:15:55.474340+0000","last_undegraded":"2024-09-22T03:15:55.474340+0000","last_fullsized":"2024-09-22T03:15:55.474340+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.470640+0000","last_change":"2024-09-22T03:15:55.473205+0000","last_active":"2024-09-22T03:15:55.470640+0000","last_peered":"2024-09-22T03:15:55.470640+0000","last_clean":"2024-09-22T03:15:55.470640+0000","last_became_active":"2024-09-22T03:15:53.577075+0000","last_became_peered":"2024-09-22T03:15:53.577075+0000","last_unstale":"2024-09-22T03:15:55.470640+0000","last_undegraded":"2024-09-22T03:15:55.470640+0000","last_fullsized":"2024-09-22T03:15:55.470640+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":16,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.468883+0000","last_change":"2024-09-22T03:15:55.474940+0000","last_active":"2024-09-22T03:15:55.468883+0000","last_peered":"2024-09-22T03:15:55.468883+0000","last_clean":"2024-09-22T03:15:55.468883+0000","last_became_active":"2024-09-22T03:15:53.561718+0000","last_became_peered":"2024-09-22T03:15:53.561718+0000","last_unstale":"2024-09-22T03:15:55.468883+0000","last_undegraded":"2024-09-22T03:15:55.468883+0000","last_fullsized":"2024-09-22T03:15:55.468883+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-22T03:15:55.469731+0000","last_change":"2024-09-22T03:15:52.494206+0000","last_active":"2024-09-22T03:15:55.469731+0000","last_peered":"2024-09-22T03:15:55.469731+0000","last_clean":"2024-09-22T03:15:55.469731+0000","last_became_active":"2024-09-22T03:15:52.482270+0000","last_became_peered":"2024-09-22T03:15:52.482270+0000","last_unstale":"2024-09-22T03:15:55.469731+0000","last_undegraded":"2024-09-22T03:15:55.469731+0000","last_fullsized":"2024-09-22T03:15:55.469731+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_clean_scrub_stamp":"2024-09-22T03:15:51.441048+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655216640,"internally_reserved":0,"allocated":3403776,"data_stored":3403776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1580,"kb_used_data":1580,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296148,"statfs":{"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":420,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297308,"statfs":{"total":53552873472,"available":53552443392,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1324,"kb_used_data":1324,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296404,"statfs":{"total":53552873472,"available":53551517696,"internally_reserved":0,"allocated":1355776,"data_stored":1355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551517696,"internally_reserved":0,"allocated":1355776,"data_stored":1355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551255552,"internally_reserved":0,"allocated":1617920,"data_stored":1617920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552443392,"internally_reserved":0,"allocated":430080,"data_stored":430080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:15:58.805 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:15:58.805 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T03:15:58.805 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T03:15:58.805 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T03:15:58.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:58.902+0000 7fcf2a82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:58.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:15:58.907+0000 7fcf2a82c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:15:59.118 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:15:59.119 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T03:15:59.131 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T03:15:59.131 INFO:teuthology.run_tasks:Running task workunit... 2024-09-22T03:15:59.143 INFO:tasks.workunit:Pulling workunits from ref 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:15:59.143 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-22T03:15:59.143 DEBUG:teuthology.orchestra.run.smithi070:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-22T03:15:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:15:59.159 INFO:teuthology.orchestra.run.smithi070.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-22T03:15:59.159 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-22T03:15:59.213 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-22T03:15:59.213 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-22T03:15:59.268 INFO:tasks.workunit:timeout=3h 2024-09-22T03:15:59.268 INFO:tasks.workunit:cleanup=True 2024-09-22T03:15:59.269 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:15:59.324 INFO:tasks.workunit.client.0.smithi070.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr:Note: switching to '4e45e6b490e6cbe082cdf155fea780438ec53fc7'. 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr:state without impacting any branches by switching back to a branch. 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-22T03:16:55.842 INFO:tasks.workunit.client.0.smithi070.stderr: git switch -c 2024-09-22T03:16:55.843 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-22T03:16:55.843 INFO:tasks.workunit.client.0.smithi070.stderr:Or undo this operation with: 2024-09-22T03:16:55.843 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-22T03:16:55.843 INFO:tasks.workunit.client.0.smithi070.stderr: git switch - 2024-09-22T03:16:55.843 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-22T03:16:55.843 INFO:tasks.workunit.client.0.smithi070.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-22T03:16:55.843 INFO:tasks.workunit.client.0.smithi070.stderr: 2024-09-22T03:16:55.843 INFO:tasks.workunit.client.0.smithi070.stderr:HEAD is now at 4e45e6b490e Merge pull request #59907 from zdover23/wip-doc-2024-09-21-backport-59904-to-squid 2024-09-22T03:16:55.847 DEBUG:teuthology.orchestra.run.smithi070:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-22T03:16:55.959 INFO:tasks.workunit.client.0.smithi070.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-22T03:16:55.962 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-22T03:16:55.962 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-22T03:16:56.584 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-22T03:16:56.657 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-22T03:16:56.696 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-22T03:16:56.698 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-22T03:16:56.698 INFO:tasks.workunit.client.0.smithi070.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-22T03:16:56.736 INFO:tasks.workunit.client.0.smithi070.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-22T03:16:56.739 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-22T03:16:56.739 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-22T03:16:56.755 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2024-09-22T03:16:56.756 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2024-09-22T03:16:56.756 DEBUG:teuthology.orchestra.run.smithi070:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-09-22T03:16:56.852 INFO:tasks.workunit.client.0.smithi070.stdout:seed 29639 2024-09-22T03:16:56.852 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = TestLibRBD.* 2024-09-22T03:16:56.852 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 170 tests from 1 test suite. 2024-09-22T03:16:56.852 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:16:56.852 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 170 tests from TestLibRBD 2024-09-22T03:16:56.890 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-09-22T03:16:58.463 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:00.642 INFO:tasks.workunit.client.0.smithi070.stdout:image has size 2097152 and order 22 2024-09-22T03:17:00.642 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CreateAndStat (3752 ms) 2024-09-22T03:17:00.643 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-09-22T03:17:00.643 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:00.765 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (124 ms) 2024-09-22T03:17:00.766 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-09-22T03:17:00.766 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:00.883 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CreateAndStatPP (117 ms) 2024-09-22T03:17:00.883 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetId 2024-09-22T03:17:00.883 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:01.056 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetId (173 ms) 2024-09-22T03:17:01.056 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-09-22T03:17:01.056 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:01.225 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetIdPP (169 ms) 2024-09-22T03:17:01.225 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-09-22T03:17:01.226 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:01.343 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (118 ms) 2024-09-22T03:17:01.343 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-09-22T03:17:01.343 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:01.459 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (116 ms) 2024-09-22T03:17:01.459 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-09-22T03:17:01.459 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:01.571 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (112 ms) 2024-09-22T03:17:01.571 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-09-22T03:17:01.571 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:01.684 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (113 ms) 2024-09-22T03:17:01.684 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.OpenAio 2024-09-22T03:17:01.684 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:01.803 INFO:tasks.workunit.client.0.smithi070.stdout:image has size 2097152 and order 22 2024-09-22T03:17:01.803 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.OpenAio (119 ms) 2024-09-22T03:17:01.803 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-09-22T03:17:01.811 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.OpenAioFail (8 ms) 2024-09-22T03:17:01.811 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-09-22T03:17:01.812 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:02.004 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.OpenAioPP (193 ms) 2024-09-22T03:17:02.004 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-09-22T03:17:02.012 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.OpenAioFailPP (8 ms) 2024-09-22T03:17:02.012 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-09-22T03:17:02.012 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:02.318 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ResizeAndStat (306 ms) 2024-09-22T03:17:02.318 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-09-22T03:17:02.318 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:02.567 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (249 ms) 2024-09-22T03:17:02.567 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-09-22T03:17:02.567 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:02.847 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (280 ms) 2024-09-22T03:17:02.847 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-09-22T03:17:02.847 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:03.135 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (288 ms) 2024-09-22T03:17:03.135 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-09-22T03:17:04.595 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:06.613 INFO:tasks.workunit.client.0.smithi070.stdout:image: image18 2024-09-22T03:17:06.613 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image18 2024-09-22T03:17:06.613 INFO:tasks.workunit.client.0.smithi070.stdout:found image18 2024-09-22T03:17:06.613 INFO:tasks.workunit.client.0.smithi070.stdout:erased image18 2024-09-22T03:17:06.613 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:image: image18 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:image: image19 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image18 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:found image18 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:erased image18 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image19 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:found image19 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:erased image19 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:image: image19 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image19 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:found image19 2024-09-22T03:17:07.196 INFO:tasks.workunit.client.0.smithi070.stdout:erased image19 2024-09-22T03:17:07.197 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (4061 ms) 2024-09-22T03:17:07.197 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-09-22T03:17:08.493 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:10.613 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 1 2024-09-22T03:17:10.613 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:17:10.613 INFO:tasks.workunit.client.0.smithi070.stdout:image: image20 2024-09-22T03:17:10.613 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image20 2024-09-22T03:17:10.613 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:10.686 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 2 2024-09-22T03:17:10.686 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:17:10.686 INFO:tasks.workunit.client.0.smithi070.stdout:image: image20 2024-09-22T03:17:10.686 INFO:tasks.workunit.client.0.smithi070.stdout:image: image21 2024-09-22T03:17:10.686 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image20 2024-09-22T03:17:10.687 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image21 2024-09-22T03:17:11.139 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 1 2024-09-22T03:17:11.139 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:17:11.140 INFO:tasks.workunit.client.0.smithi070.stdout:image: image21 2024-09-22T03:17:11.140 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image21 2024-09-22T03:17:11.140 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (3943 ms) 2024-09-22T03:17:11.140 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCopy 2024-09-22T03:17:12.508 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:18.105 INFO:tasks.workunit.client.0.smithi070.stdout:image: image22 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image22 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:found image22 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:erased image22 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:image: image22 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:image: image23 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image22 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:found image22 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:erased image22 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image23 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:found image23 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:erased image23 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-22T03:17:18.106 INFO:tasks.workunit.client.0.smithi070.stdout:image: image22 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:image: image23 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:image: image24 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image22 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:found image22 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:erased image22 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image23 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:found image23 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:erased image23 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image24 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:found image24 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:erased image24 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCopy (6966 ms) 2024-09-22T03:17:18.107 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-09-22T03:17:19.523 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:24.228 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 1 2024-09-22T03:17:24.228 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:17:24.228 INFO:tasks.workunit.client.0.smithi070.stdout:image: image25 2024-09-22T03:17:24.228 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image25 2024-09-22T03:17:24.371 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 2 2024-09-22T03:17:24.372 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:17:24.372 INFO:tasks.workunit.client.0.smithi070.stdout:image: image25 2024-09-22T03:17:24.372 INFO:tasks.workunit.client.0.smithi070.stdout:image: image26 2024-09-22T03:17:24.372 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image25 2024-09-22T03:17:24.372 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image26 2024-09-22T03:17:24.831 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-22T03:17:24.831 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 3 2024-09-22T03:17:24.831 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-22T03:17:24.831 INFO:tasks.workunit.client.0.smithi070.stdout:image: image25 2024-09-22T03:17:24.831 INFO:tasks.workunit.client.0.smithi070.stdout:image: image26 2024-09-22T03:17:24.831 INFO:tasks.workunit.client.0.smithi070.stdout:image: image27 2024-09-22T03:17:24.831 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image25 2024-09-22T03:17:24.832 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image26 2024-09-22T03:17:24.832 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image27 2024-09-22T03:17:25.166 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCopyPP (7061 ms) 2024-09-22T03:17:25.166 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-09-22T03:17:26.538 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-22T03:17:36.132 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:image: image30 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image30 2024-09-22T03:17:36.133 INFO:tasks.workunit.client.0.smithi070.stdout:found image30 2024-09-22T03:17:36.134 INFO:tasks.workunit.client.0.smithi070.stdout:erased image30 2024-09-22T03:17:36.134 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-22T03:17:36.134 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-22T03:17:36.134 INFO:tasks.workunit.client.0.smithi070.stdout:image: image30 2024-09-22T03:17:36.134 INFO:tasks.workunit.client.0.smithi070.stdout:image: image31 2024-09-22T03:17:36.134 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image30 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:found image30 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:erased image30 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image31 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:found image31 2024-09-22T03:17:36.135 INFO:tasks.workunit.client.0.smithi070.stdout:erased image31 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:image: image30 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:image: image31 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:image: image32 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image30 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:found image30 2024-09-22T03:17:36.136 INFO:tasks.workunit.client.0.smithi070.stdout:erased image30 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image31 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:found image31 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:erased image31 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image32 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:found image32 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:erased image32 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:image: image28 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:image: image29 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:image: image30 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:image: image31 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:image: image32 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:image: image33 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image28 2024-09-22T03:17:36.137 INFO:tasks.workunit.client.0.smithi070.stdout:found image28 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:erased image28 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image29 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:found image29 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:erased image29 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image30 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:found image30 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:erased image30 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image31 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:found image31 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:erased image31 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image32 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:found image32 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:erased image32 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image33 2024-09-22T03:17:36.138 INFO:tasks.workunit.client.0.smithi070.stdout:found image33 2024-09-22T03:17:36.139 INFO:tasks.workunit.client.0.smithi070.stdout:erased image33 2024-09-22T03:17:36.139 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestDeepCopy (10966 ms) 2024-09-22T03:17:36.139 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-09-22T03:17:37.483 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:42.327 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 1 2024-09-22T03:17:42.327 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:17:42.327 INFO:tasks.workunit.client.0.smithi070.stdout:image: image34 2024-09-22T03:17:42.327 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image34 2024-09-22T03:17:42.541 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 2 2024-09-22T03:17:42.541 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:17:42.541 INFO:tasks.workunit.client.0.smithi070.stdout:image: image34 2024-09-22T03:17:42.541 INFO:tasks.workunit.client.0.smithi070.stdout:image: image35 2024-09-22T03:17:42.541 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image34 2024-09-22T03:17:42.541 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image35 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:num images is: 3 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:image: image34 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:image: image35 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:image: image36 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image34 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image35 2024-09-22T03:17:43.083 INFO:tasks.workunit.client.0.smithi070.stdout:expected = image36 2024-09-22T03:17:43.412 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (7280 ms) 2024-09-22T03:17:43.412 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-09-22T03:17:43.412 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-22T03:17:47.656 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-22T03:17:47.657 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-22T03:17:47.657 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:17:47.657 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (4243 ms) 2024-09-22T03:17:47.657 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-09-22T03:17:47.657 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:49.700 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (2046 ms) 2024-09-22T03:17:49.701 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-09-22T03:17:49.701 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:50.652 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-22T03:17:50.652 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:17:50.652 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-22T03:17:50.652 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-22T03:17:51.656 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-22T03:17:51.656 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:17:51.656 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-22T03:17:51.657 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-22T03:17:51.657 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-22T03:17:51.657 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-22T03:17:52.664 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-22T03:17:52.664 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:17:52.664 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-22T03:17:52.664 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-22T03:17:53.643 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-22T03:17:53.643 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:17:53.677 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (3976 ms) 2024-09-22T03:17:53.677 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-09-22T03:17:53.677 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:17:59.673 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-22T03:17:59.673 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:17:59.707 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6030 ms) 2024-09-22T03:17:59.708 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-09-22T03:17:59.708 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:00.737 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-22T03:18:00.737 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:18:00.737 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-22T03:18:00.737 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-22T03:18:01.708 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-22T03:18:01.708 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:18:01.708 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1 2024-09-22T03:18:01.708 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-22T03:18:01.708 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1 with size 2097152 2024-09-22T03:18:01.708 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-22T03:18:01.761 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-22T03:18:01.761 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:18:01.761 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap1-rename 2024-09-22T03:18:01.761 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2 2024-09-22T03:18:01.761 INFO:tasks.workunit.client.0.smithi070.stdout:found snap1-rename with size 2097152 2024-09-22T03:18:01.761 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2 with size 4194304 2024-09-22T03:18:02.789 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-22T03:18:02.789 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:18:02.789 INFO:tasks.workunit.client.0.smithi070.stdout:snap: snap2-rename 2024-09-22T03:18:02.789 INFO:tasks.workunit.client.0.smithi070.stdout:found snap2-rename with size 4194304 2024-09-22T03:18:03.671 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-22T03:18:03.671 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:18:03.703 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (3996 ms) 2024-09-22T03:18:03.703 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-09-22T03:18:05.566 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:05.567 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:05.567 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:05.567 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:09.533 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (5830 ms) 2024-09-22T03:18:09.533 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-09-22T03:18:13.807 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:28.077 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RemoveFullTry (18544 ms) 2024-09-22T03:18:28.077 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-09-22T03:18:34.518 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:53.484 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:54.640 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:55.721 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:56.797 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:57.861 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:18:58.549 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RemoveFullTryDataPool (30472 ms) 2024-09-22T03:18:58.549 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-09-22T03:19:01.038 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:19:18.879 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespace (20330 ms) 2024-09-22T03:19:18.879 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-09-22T03:19:22.848 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:19:44.503 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:19:45.778 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:19:46.844 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:19:47.916 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:19:48.586 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespaceDataPool (29707 ms) 2024-09-22T03:19:48.586 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIO 2024-09-22T03:19:48.586 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:20:03.964 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.965 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:03.966 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:03.967 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:03.968 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:03.969 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.970 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.971 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:20:03.972 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.973 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.974 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:20:03.975 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.976 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:03.977 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.978 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.979 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:03.979 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:03.979 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:03.979 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:03.979 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:03.979 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:03.979 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:03.979 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.011 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.012 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.013 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.014 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.015 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:17.016 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.017 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:17.018 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIO (28425 ms) 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-09-22T03:20:17.019 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:20:38.217 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:38.218 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:38.218 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:38.218 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:38.218 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:38.218 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.218 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:38.219 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.219 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.219 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:38.219 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.219 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:38.220 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.220 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.220 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:38.220 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.220 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:38.220 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.220 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.221 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:38.221 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.221 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:38.221 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.221 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.221 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:38.222 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.222 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:20:38.222 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.222 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.222 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:20:38.223 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:38.223 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:38.223 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:38.223 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:38.223 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:38.224 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:38.224 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:38.224 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:38.224 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:20:38.224 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:20:38.224 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.224 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:38.225 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.225 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.225 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:38.225 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.225 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:38.226 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.226 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.226 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:38.226 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.226 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:38.226 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.227 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.227 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:38.227 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.227 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:38.227 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.227 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.228 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:38.228 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.228 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:20:38.228 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.228 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.228 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:20:38.229 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.229 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.229 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.229 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.229 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.229 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.230 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:38.230 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:38.230 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:38.230 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.230 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:38.230 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:38.231 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:38.231 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.231 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:38.231 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:38.231 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:38.232 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.232 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:38.232 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:38.232 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:38.232 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.232 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:20:38.233 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:38.233 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:20:38.233 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-22T03:20:38.233 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.233 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-22T03:20:38.233 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.234 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.234 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.234 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.234 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:20:38.234 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:20:38.234 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.235 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.235 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:20:38.235 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.235 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.235 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:20:38.235 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.236 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.236 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:20:38.236 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.236 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.236 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.236 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.237 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:20:38.237 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.237 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.237 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:20:38.237 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.237 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.238 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:20:38.238 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.238 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.238 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:20:38.238 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.238 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.239 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.239 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.239 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:20:38.239 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.239 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.239 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:20:38.239 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.240 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.240 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:20:38.240 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.240 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.240 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:20:38.241 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.241 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.241 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.241 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.241 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:20:38.241 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.242 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.242 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:20:38.242 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.242 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.242 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:20:38.242 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.243 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.243 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:20:38.243 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.243 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.243 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.243 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.244 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:20:38.244 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.244 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.244 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:20:38.244 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.245 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.245 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:20:38.245 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.245 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.245 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:20:38.245 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.246 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.246 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.246 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.246 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.246 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:38.246 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.247 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.247 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:38.247 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.247 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.247 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.248 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:38.248 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.248 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.248 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:38.248 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.248 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.249 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.249 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:38.249 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.249 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.249 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:38.249 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.250 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.250 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.250 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:38.250 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.250 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.250 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:38.251 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.251 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.251 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.251 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:38.251 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.252 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.252 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:38.252 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:38.252 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.252 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:38.252 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.253 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.253 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:38.253 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.253 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.253 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:38.253 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:38.254 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:38.254 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:38.254 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:38.254 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:38.254 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:38.254 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.803 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.804 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.805 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.806 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.807 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.808 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.809 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:20:57.810 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-22T03:20:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:20:57.812 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:20:57.812 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (40792 ms) 2024-09-22T03:20:57.812 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-09-22T03:20:57.812 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:21:21.352 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:21:21.352 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:21:21.352 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:21:21.352 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:21:21.352 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:21:21.352 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.352 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:21:21.352 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.353 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:21:21.354 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.355 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.356 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:21:21.357 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.358 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:21:21.359 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.360 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:21:21.361 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:21:21.362 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.363 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.364 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.365 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:21.366 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.947 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.947 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.947 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.947 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.948 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.949 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.950 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.951 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.952 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.953 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.954 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:21:39.955 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (42144 ms) 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-09-22T03:21:39.956 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:read: 4 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCloneEncryption (65477 ms) 2024-09-22T03:22:45.424 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-09-22T03:22:45.425 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:23:01.039 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (15615 ms) 2024-09-22T03:23:01.039 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-09-22T03:23:01.039 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:23:15.663 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (14624 ms) 2024-09-22T03:23:15.663 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-09-22T03:23:15.663 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:23:20.138 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4476 ms) 2024-09-22T03:23:20.139 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-09-22T03:23:20.139 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:23:26.296 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (6157 ms) 2024-09-22T03:23:26.297 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-22T03:23:26.297 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:2905: Skipped 2024-09-22T03:23:26.297 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:23:26.297 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (1 ms) 2024-09-22T03:23:26.297 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-09-22T03:23:26.297 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:25:40.248 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptionLoadFormatMismatch (133951 ms) 2024-09-22T03:25:40.248 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-09-22T03:25:40.248 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:25:52.790 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptedResize (12542 ms) 2024-09-22T03:25:52.790 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-09-22T03:25:52.790 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:06.501 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EncryptedFlattenSmallData (13711 ms) 2024-09-22T03:26:06.501 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-09-22T03:26:06.501 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:22.723 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:22.723 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:22.723 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:22.723 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:22.724 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:22.725 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.726 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.727 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:22.728 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:22.729 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.730 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.731 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.732 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:26:22.733 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:26:22.734 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.735 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.736 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.737 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:22.738 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:22.739 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:22.739 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:22.739 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:22.739 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:22.739 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:22.739 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.533 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.534 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.535 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.536 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.537 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.538 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.539 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.540 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.541 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (30033 ms) 2024-09-22T03:26:36.542 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-09-22T03:26:38.447 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.031 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.032 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:41.033 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:41.034 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 = 512 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 1536~512 = 0 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestDataPoolIO (4497 ms) 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-09-22T03:26:41.035 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:41.262 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (231 ms) 2024-09-22T03:26:41.262 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-09-22T03:26:41.262 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:41.493 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (231 ms) 2024-09-22T03:26:41.493 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-09-22T03:26:41.493 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:41.736 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (244 ms) 2024-09-22T03:26:41.737 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-09-22T03:26:41.737 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (248 ms) 2024-09-22T03:26:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-09-22T03:26:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:42.415 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (431 ms) 2024-09-22T03:26:42.416 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-09-22T03:26:42.416 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:42.926 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (511 ms) 2024-09-22T03:26:42.926 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-09-22T03:26:42.927 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:43.316 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (389 ms) 2024-09-22T03:26:43.316 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-09-22T03:26:43.316 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:43.773 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (457 ms) 2024-09-22T03:26:43.773 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-09-22T03:26:43.773 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:46.043 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (2271 ms) 2024-09-22T03:26:46.044 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-09-22T03:26:46.044 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:48.596 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (2552 ms) 2024-09-22T03:26:48.596 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-09-22T03:26:48.596 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:48.877 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (281 ms) 2024-09-22T03:26:48.877 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-09-22T03:26:48.877 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:49.125 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (249 ms) 2024-09-22T03:26:49.126 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-09-22T03:26:49.126 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:49.361 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (235 ms) 2024-09-22T03:26:49.361 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-09-22T03:26:49.361 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:49.583 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (222 ms) 2024-09-22T03:26:49.583 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-09-22T03:26:49.583 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:49.806 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:49.807 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 0~1048576 = 0 2024-09-22T03:26:49.807 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:49.807 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 0~4194304 = 0 2024-09-22T03:26:49.807 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:49.807 INFO:tasks.workunit.client.0.smithi070.stdout:aio discard: 3145728~1048576 = 0 2024-09-22T03:26:49.807 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (224 ms) 2024-09-22T03:26:49.807 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestFUA 2024-09-22T03:26:49.807 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:50.839 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:50.840 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.841 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:50.842 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:50.843 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:50.843 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:50.843 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:50.843 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestFUA (1032 ms) 2024-09-22T03:26:50.843 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-09-22T03:26:50.843 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:26:51.014 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:51.014 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:51.014 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:51.014 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:51.014 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:26:51.014 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.014 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:51.015 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.016 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:51.016 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.016 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.016 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:51.016 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.016 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:26:51.016 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:51.175 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:51.176 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:51.176 INFO:tasks.workunit.client.0.smithi070.stdout:start compare and write 2024-09-22T03:26:51.176 INFO:tasks.workunit.client.0.smithi070.stdout:compare and wrote: 512 2024-09-22T03:26:51.176 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.176 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:51.176 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.185 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.185 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:51.185 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.185 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:51.186 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.196 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.196 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:51.196 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.196 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:51.196 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.207 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.207 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:51.207 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.207 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:51.208 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.218 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.219 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:51.219 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.219 INFO:tasks.workunit.client.0.smithi070.stdout:started aio compare and write 2024-09-22T03:26:51.219 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:26:51.236 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:26:51.236 INFO:tasks.workunit.client.0.smithi070.stdout:finished aio compare and write 2024-09-22T03:26:51.236 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:51.237 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:51.237 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:51.237 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:51.237 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:26:51.237 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.237 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:51.237 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:51.239 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:51.239 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:26:51.239 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.239 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:51.239 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:51.242 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:51.242 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:26:51.242 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.242 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:51.242 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:51.244 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:51.245 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:26:51.245 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.245 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:51.245 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:51.247 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:51.247 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:26:51.247 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:26:51.247 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:26:51.247 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:26:51.248 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:26:51.248 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:26:51.248 INFO:tasks.workunit.client.0.smithi070.stdout:discard: 512~512 2024-09-22T03:26:51.248 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:02.114 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:27:02.115 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.116 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:27:02.117 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:27:02.118 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.119 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:02.120 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.121 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.121 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.121 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.121 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:02.217 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:02.217 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:02.217 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.217 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.217 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.217 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.218 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.288 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:02.582 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:02.582 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:02.582 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.582 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.583 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.583 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.583 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:02.795 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:02.795 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:02.795 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:02.796 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:02.796 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:02.796 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:02.796 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:03.215 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:03.215 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:03.215 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:03.215 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:03.216 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:03.216 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:03.216 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:03.464 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:03.465 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:04.098 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:04.098 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:04.098 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:04.098 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:04.099 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:04.099 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:04.099 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:04.476 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:04.476 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:04.476 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:04.476 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:04.476 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:04.476 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:04.476 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:05.234 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:05.234 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:05.234 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:05.235 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:05.235 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:05.235 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:05.235 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:05.663 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:05.663 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:05.663 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:05.663 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:05.663 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:05.664 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:05.664 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:05.664 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:05.664 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:05.664 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:05.664 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:05.664 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:05.664 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:06.566 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:06.566 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:06.566 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:06.566 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:06.566 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:06.566 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:06.566 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:07.172 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:07.172 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:07.172 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:07.172 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:07.172 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:07.172 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:07.172 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:08.133 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:08.133 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:08.133 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:08.133 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:08.134 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:08.134 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:08.134 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:08.754 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:08.755 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:08.755 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:08.755 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:08.755 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:10.011 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:10.011 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:10.012 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:10.012 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:10.012 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:10.012 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:10.012 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:10.774 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:10.774 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:10.774 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:10.774 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:10.774 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:10.774 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:10.774 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:12.098 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:12.098 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:12.098 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:12.098 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:12.098 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:12.098 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:12.098 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIOPP (22161 ms) 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-09-22T03:27:13.000 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:13.224 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (225 ms) 2024-09-22T03:27:13.224 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-09-22T03:27:13.224 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:13.431 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (207 ms) 2024-09-22T03:27:13.432 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-09-22T03:27:13.432 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:13.649 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (217 ms) 2024-09-22T03:27:13.649 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-09-22T03:27:13.649 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:13.880 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (232 ms) 2024-09-22T03:27:13.881 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-09-22T03:27:13.881 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:14.120 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (240 ms) 2024-09-22T03:27:14.121 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-09-22T03:27:14.121 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:14.293 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:14.333 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (213 ms) 2024-09-22T03:27:14.333 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-22T03:27:14.334 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:14.561 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (227 ms) 2024-09-22T03:27:14.561 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-22T03:27:14.561 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:14.758 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:14.806 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (246 ms) 2024-09-22T03:27:14.807 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-09-22T03:27:14.807 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:15.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (292 ms) 2024-09-22T03:27:15.099 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-09-22T03:27:15.099 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:15.292 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:15.332 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (233 ms) 2024-09-22T03:27:15.332 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-22T03:27:15.332 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:15.577 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (245 ms) 2024-09-22T03:27:15.577 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-22T03:27:15.577 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:15.771 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:15.815 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (238 ms) 2024-09-22T03:27:15.815 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-09-22T03:27:15.815 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:16.212 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (397 ms) 2024-09-22T03:27:16.212 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-09-22T03:27:16.212 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:16.507 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:16.877 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (665 ms) 2024-09-22T03:27:16.877 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-09-22T03:27:16.877 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:17.278 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (401 ms) 2024-09-22T03:27:17.278 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-09-22T03:27:17.278 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:17.516 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:17.658 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (380 ms) 2024-09-22T03:27:17.658 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-09-22T03:27:17.658 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:20.253 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (2596 ms) 2024-09-22T03:27:20.253 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-09-22T03:27:20.253 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:22.066 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:22.281 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (2027 ms) 2024-09-22T03:27:22.281 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-09-22T03:27:22.281 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:22.499 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:27:22.499 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:27:22.499 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:27:22.499 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:27:22.499 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:27:22.500 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.500 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:27:22.500 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:22.509 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:22.509 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:27:22.509 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.509 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:27:22.509 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:22.518 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:22.518 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:27:22.518 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.518 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:27:22.519 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:22.528 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:22.528 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:27:22.528 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.528 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:27:22.528 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:22.538 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:22.538 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:27:22.538 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.538 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:27:22.538 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:22.544 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:22.544 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:27:22.544 INFO:tasks.workunit.client.0.smithi070.stdout:read: 512 2024-09-22T03:27:22.544 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.544 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:27:22.545 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:27:22.547 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:27:22.547 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:27:22.547 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.547 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:27:22.547 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:27:22.549 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:27:22.549 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:27:22.549 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.550 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:27:22.550 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:27:22.552 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:27:22.552 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:27:22.552 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.552 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:27:22.552 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:27:22.555 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:27:22.555 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:27:22.555 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:22.555 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:27:22.555 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:27:33.502 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:27:33.502 INFO:tasks.workunit.client.0.smithi070.stdout:finished read 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 0 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 16384 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.503 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 49152 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 65536 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.504 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 98304 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 114688 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.505 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 147456 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.506 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 163840 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 196608 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:27:33.507 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 212992 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:33.508 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.509 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.509 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:33.509 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:33.509 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:33.512 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:33.512 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:33.512 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.512 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.512 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:33.513 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:33.513 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:33.614 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:33.614 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:33.615 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.615 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.615 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:33.615 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:33.615 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:33.699 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:33.699 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:33.699 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:33.699 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:33.699 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:33.699 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:33.700 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.700 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:33.700 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:33.700 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:33.700 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:33.700 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:33.700 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:34.090 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:34.090 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:34.090 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:34.091 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:34.091 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:34.091 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:34.091 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:34.276 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:34.277 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:34.277 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:34.277 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:34.277 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:34.277 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:34.277 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:34.659 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:34.659 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:34.659 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:34.660 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:34.660 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:34.660 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:34.660 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:34.969 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:34.969 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:34.969 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:34.969 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:34.969 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:34.969 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:34.970 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:34.970 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:34.970 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:34.970 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:34.970 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:34.970 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:34.970 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:35.550 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:35.550 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:35.550 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:35.550 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:35.550 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:35.550 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:35.550 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:35.968 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:35.968 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:35.968 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:35.969 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:35.969 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:35.969 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:35.969 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:36.664 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:36.664 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:36.664 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:36.664 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:36.664 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:36.664 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:36.664 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:37.196 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:37.197 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:38.106 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:38.106 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:38.107 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:38.107 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:38.107 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:38.107 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:38.107 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:38.653 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:38.653 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:38.653 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:38.653 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:38.654 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:38.654 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:38.654 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:39.717 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:39.717 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:39.717 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:39.717 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:39.717 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:39.717 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:39.717 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:40.311 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:40.312 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:40.313 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:40.313 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:40.313 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:41.510 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:41.510 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:41.510 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:41.510 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:41.510 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:41.510 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:41.510 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:42.317 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:42.317 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:42.317 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:42.317 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:42.317 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:42.317 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:42.317 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:43.612 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:43.612 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:43.612 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:43.612 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:43.612 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:43.612 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:43.612 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:finished writesame 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:to verify the data 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:verified 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:27:44.504 INFO:tasks.workunit.client.0.smithi070.stdout:started writesame 2024-09-22T03:27:44.505 INFO:tasks.workunit.client.0.smithi070.stdout:expected fail, finished writesame 2024-09-22T03:27:44.505 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (22223 ms) 2024-09-22T03:27:44.505 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-09-22T03:27:44.505 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:46.797 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-22T03:27:46.797 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:27:46.797 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-22T03:27:46.797 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:27:46.797 INFO:tasks.workunit.client.0.smithi070.stdout:snap: orig 2024-09-22T03:27:46.797 INFO:tasks.workunit.client.0.smithi070.stdout:found orig with size 2097152 2024-09-22T03:27:46.797 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-22T03:27:46.797 INFO:tasks.workunit.client.0.smithi070.stdout:write test data! 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 80 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:snap: orig 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:snap: written 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:found orig with size 2097152 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:found written with size 2097152 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:write to snapshot returned -30 2024-09-22T03:27:46.798 INFO:tasks.workunit.client.0.smithi070.stdout:Read-only file system 2024-09-22T03:27:47.093 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-22T03:27:47.094 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-22T03:27:47.094 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 80 2024-09-22T03:27:47.094 INFO:tasks.workunit.client.0.smithi070.stdout:opening testimg@orig 2024-09-22T03:27:47.094 INFO:tasks.workunit.client.0.smithi070.stdout:read: 80 2024-09-22T03:27:47.094 INFO:tasks.workunit.client.0.smithi070.stdout:write to snapshot returned -30 2024-09-22T03:27:47.094 INFO:tasks.workunit.client.0.smithi070.stdout:Read-only file system 2024-09-22T03:27:48.754 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 2 2024-09-22T03:27:48.754 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:27:48.754 INFO:tasks.workunit.client.0.smithi070.stdout:snap: orig 2024-09-22T03:27:48.754 INFO:tasks.workunit.client.0.smithi070.stdout:snap: written 2024-09-22T03:27:48.754 INFO:tasks.workunit.client.0.smithi070.stdout:found orig with size 2097152 2024-09-22T03:27:48.754 INFO:tasks.workunit.client.0.smithi070.stdout:found written with size 2097152 2024-09-22T03:27:48.754 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 1 2024-09-22T03:27:48.754 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:27:48.755 INFO:tasks.workunit.client.0.smithi070.stdout:snap: orig 2024-09-22T03:27:48.755 INFO:tasks.workunit.client.0.smithi070.stdout:found orig with size 2097152 2024-09-22T03:27:48.755 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-22T03:27:48.755 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:27:48.755 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (4250 ms) 2024-09-22T03:27:48.755 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-09-22T03:27:48.755 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:50.727 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (1973 ms) 2024-09-22T03:27:50.727 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestClone 2024-09-22T03:27:50.727 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:27:57.676 INFO:tasks.workunit.client.0.smithi070.stdout:made parent image "parent" 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:parent has no parent info 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:made snapshot "parent@parent_snap" 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:can't unprotect an unprotected snap 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:can't protect a protected snap 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:made and opened clone "child" 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:read: 8 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:read: 8 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:read: 8 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:sizes and overlaps are good between parent and child 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:child image successfully cloned all image-meta pairs 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:sized down clone, changed overlap 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:parent info: size 4194304 obj_size 4194304 parent_pool 18446744073709551615 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:sized up clone, changed size but not overlap or parent's size 2024-09-22T03:27:57.677 INFO:tasks.workunit.client.0.smithi070.stdout:can't remove parent while child still exists 2024-09-22T03:27:57.678 INFO:tasks.workunit.client.0.smithi070.stdout:can't remove parent while still protected 2024-09-22T03:27:57.678 INFO:tasks.workunit.client.0.smithi070.stdout:removed parent snap after unprotecting 2024-09-22T03:27:57.678 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestClone (6950 ms) 2024-09-22T03:27:57.678 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestClone2 2024-09-22T03:27:57.678 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:02.870 INFO:tasks.workunit.client.0.smithi070.stdout:made parent image "parent" 2024-09-22T03:28:02.870 INFO:tasks.workunit.client.0.smithi070.stdout:parent has no parent info 2024-09-22T03:28:02.871 INFO:tasks.workunit.client.0.smithi070.stdout:made snapshot "parent@parent_snap" 2024-09-22T03:28:02.871 INFO:tasks.workunit.client.0.smithi070.stdout:made and opened clone "child" 2024-09-22T03:28:02.871 INFO:tasks.workunit.client.0.smithi070.stdout:child image successfully cloned all image-meta pairs 2024-09-22T03:28:02.871 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestClone2 (5193 ms) 2024-09-22T03:28:02.871 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ListChildren 2024-09-22T03:28:06.540 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 1 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout:child: image124 2024-09-22T03:28:16.307 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124/1060952c61f3 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 2 2024-09-22T03:28:16.308 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:child: image124 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124/1060952c61f3 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125/10605f698137 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.309 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-22T03:28:16.310 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:child: image124 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124/1060952c61f3 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125/10605f698137 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image126/10605d5c1091 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.311 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:child: image124 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124/1060952c61f3 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125/10605f698137 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image126/10605d5c1091 2024-09-22T03:28:16.312 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.313 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 4 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 4 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:child: image124 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-22T03:28:16.314 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124/1060952c61f3 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125/10605f698137 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image126/10605d5c1091 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image127/10601658c288 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.315 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:16.316 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.404 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image124 2024-09-22T03:28:21.404 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:21.404 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:21.404 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:21.404 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.404 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 4 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 4 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:child: image124 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image124/1060952c61f3 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125/10605f698137 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image126/10605d5c1091 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image127/10601658c288 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.405 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.406 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image126 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:child: image126 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125/10605f698137 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image126/10605d5c1091 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image127/10601658c288 2024-09-22T03:28:21.407 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image127 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 2 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 2 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-22T03:28:21.408 INFO:tasks.workunit.client.0.smithi070.stdout:child: image127 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125/10605f698137 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-17/image127/10601658c288 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:trying to find test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:checking test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 1 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 1 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:child: image125 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-16/image125/10605f698137 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:28:21.409 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 0 2024-09-22T03:28:21.410 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:28:21.410 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ListChildren (18534 ms) 2024-09-22T03:28:21.410 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-09-22T03:28:21.410 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:6995: Skipped 2024-09-22T03:28:21.410 INFO:tasks.workunit.client.0.smithi070.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:28:21.410 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered (0 ms) 2024-09-22T03:28:21.410 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.LockingPP 2024-09-22T03:28:21.410 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:21.834 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.LockingPP (431 ms) 2024-09-22T03:28:21.835 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlushAio 2024-09-22T03:28:21.835 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:24.842 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlushAio (3007 ms) 2024-09-22T03:28:24.842 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-09-22T03:28:24.842 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:27.891 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlushAioPP (3049 ms) 2024-09-22T03:28:27.891 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-09-22T03:28:27.891 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:28.086 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (195 ms) 2024-09-22T03:28:28.086 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-09-22T03:28:28.086 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:28.303 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (217 ms) 2024-09-22T03:28:28.303 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-09-22T03:28:28.303 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:28.433 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ZeroLengthRead (130 ms) 2024-09-22T03:28:28.433 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-09-22T03:28:28.433 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:28.732 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.LargeCacheRead (299 ms) 2024-09-22T03:28:28.732 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-09-22T03:28:28.732 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:31.733 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestPendingAio (3001 ms) 2024-09-22T03:28:31.733 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-09-22T03:28:31.733 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:33.372 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:35.818 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyup (4085 ms) 2024-09-22T03:28:35.818 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-09-22T03:28:35.818 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:37.353 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:39.828 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyupMismatch (4010 ms) 2024-09-22T03:28:39.828 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Flatten 2024-09-22T03:28:39.828 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:41.388 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Flatten (1560 ms) 2024-09-22T03:28:41.388 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Sparsify 2024-09-22T03:28:41.388 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:41.697 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 32768 2024-09-22T03:28:41.697 INFO:tasks.workunit.client.0.smithi070.stdout:read: 32768 2024-09-22T03:28:41.698 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Sparsify (310 ms) 2024-09-22T03:28:41.698 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-09-22T03:28:41.698 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:42.055 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SparsifyPP (358 ms) 2024-09-22T03:28:42.056 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-09-22T03:28:42.056 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:46.694 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapshotLimit (4638 ms) 2024-09-22T03:28:46.694 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-09-22T03:28:46.694 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:51.764 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5070 ms) 2024-09-22T03:28:51.764 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-09-22T03:28:51.764 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:52.266 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-22T03:28:52.266 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RebuildObjectMapViaLockOwner (503 ms) 2024-09-22T03:28:52.267 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-09-22T03:28:52.267 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:53.097 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RenameViaLockOwner (831 ms) 2024-09-22T03:28:53.098 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-09-22T03:28:53.098 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:53.917 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapCreateViaLockOwner (819 ms) 2024-09-22T03:28:53.917 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-09-22T03:28:53.917 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:55.810 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapRemoveViaLockOwner (1894 ms) 2024-09-22T03:28:55.810 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-09-22T03:28:55.811 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:56.275 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.UpdateFeaturesViaLockOwner (464 ms) 2024-09-22T03:28:56.275 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-22T03:28:56.275 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:9736: Skipped 2024-09-22T03:28:56.275 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:28:56.275 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-09-22T03:28:56.275 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-09-22T03:28:56.275 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:58.767 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-22T03:28:58.767 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapRemove2 (2493 ms) 2024-09-22T03:28:58.767 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-09-22T03:28:58.768 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:28:59.937 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapRenameViaLockOwner (1170 ms) 2024-09-22T03:28:59.937 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-09-22T03:28:59.938 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:01.001 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapProtectViaLockOwner (1064 ms) 2024-09-22T03:29:01.002 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-09-22T03:29:01.002 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:02.146 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapUnprotectViaLockOwner (1145 ms) 2024-09-22T03:29:02.147 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-09-22T03:29:02.147 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:03.622 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlattenViaLockOwner (1475 ms) 2024-09-22T03:29:03.622 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-09-22T03:29:03.622 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:04.062 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ResizeViaLockOwner (441 ms) 2024-09-22T03:29:04.062 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-09-22T03:29:04.063 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:04.414 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SparsifyViaLockOwner (351 ms) 2024-09-22T03:29:04.414 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-09-22T03:29:04.414 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:23.228 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ObjectMapConsistentSnap (18814 ms) 2024-09-22T03:29:23.228 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Metadata 2024-09-22T03:29:23.228 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:25.691 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Metadata (2463 ms) 2024-09-22T03:29:25.691 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-09-22T03:29:25.691 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:27.671 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.MetadataPP (1980 ms) 2024-09-22T03:29:27.671 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-09-22T03:29:27.671 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:28.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.UpdateFeatures (1276 ms) 2024-09-22T03:29:28.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-09-22T03:29:28.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2024-09-22T03:29:28.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-09-22T03:29:28.947 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:30.267 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-22T03:29:30.267 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-22T03:29:30.267 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-22T03:29:30.268 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-22T03:29:30.268 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RebuildObjectMap (1321 ms) 2024-09-22T03:29:30.268 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-09-22T03:29:30.572 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-22T03:29:30.572 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-22T03:29:30.572 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-22T03:29:30.572 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-22T03:29:30.572 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.RebuildNewObjectMap (304 ms) 2024-09-22T03:29:30.572 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-09-22T03:29:30.572 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:32.270 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-22T03:29:32.271 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-22T03:29:32.271 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-22T03:29:32.271 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-22T03:29:32.271 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CheckObjectMap (1699 ms) 2024-09-22T03:29:32.271 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-09-22T03:29:32.271 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:32.489 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.BlockingAIO (218 ms) 2024-09-22T03:29:32.489 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-09-22T03:29:32.489 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:34.677 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ExclusiveLockTransition (2188 ms) 2024-09-22T03:29:34.677 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-09-22T03:29:34.677 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:10793: Skipped 2024-09-22T03:29:34.677 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:29:34.677 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-09-22T03:29:34.677 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-09-22T03:29:34.677 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:36.366 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CacheMayCopyOnWrite (1689 ms) 2024-09-22T03:29:36.366 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-09-22T03:29:36.366 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:37.758 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (1392 ms) 2024-09-22T03:29:37.758 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-09-22T03:29:39.773 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-22T03:29:39.773 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-22T03:29:39.773 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestImageOptions (2016 ms) 2024-09-22T03:29:39.774 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-09-22T03:29:41.591 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-22T03:29:41.591 INFO:tasks.workunit.client.0.smithi070.stdout:100.00% done 2024-09-22T03:29:41.591 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (1817 ms) 2024-09-22T03:29:41.591 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-09-22T03:29:41.592 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2024-09-22T03:29:41.592 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-09-22T03:29:41.592 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-09-22T03:29:41.592 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-09-22T03:29:41.592 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:41.982 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:29:41.982 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:29:41.982 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:29:41.982 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:29:41.982 INFO:tasks.workunit.client.0.smithi070.stdout:wrote: 512 2024-09-22T03:29:41.982 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:29:41.983 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:started write 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:write completion cb called! 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 0 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:finished write 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:29:41.984 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:created completion 2024-09-22T03:29:41.985 INFO:tasks.workunit.client.0.smithi070.stdout:started read 2024-09-22T03:29:41.986 INFO:tasks.workunit.client.0.smithi070.stdout:read completion cb called! 2024-09-22T03:29:41.986 INFO:tasks.workunit.client.0.smithi070.stdout:return value is: 512 2024-09-22T03:29:41.986 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ImagePollIO (391 ms) 2024-09-22T03:29:41.986 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Mirror 2024-09-22T03:29:42.023 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:42.086 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:44.182 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Mirror (2201 ms) 2024-09-22T03:29:44.182 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-09-22T03:29:44.276 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (93 ms) 2024-09-22T03:29:44.276 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-09-22T03:29:49.971 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.CreateWithMirrorEnabled (5696 ms) 2024-09-22T03:29:49.972 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-09-22T03:29:49.972 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:51.861 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (1889 ms) 2024-09-22T03:29:51.861 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-09-22T03:29:51.861 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:52.375 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-22T03:29:52.375 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-22T03:29:52.427 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-22T03:29:52.462 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-22T03:29:52.496 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-22T03:29:52.496 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-22T03:29:52.542 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-22T03:29:52.584 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-22T03:29:52.613 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-22T03:29:52.613 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-22T03:29:52.664 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-22T03:29:52.688 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-22T03:29:52.743 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-22T03:29:52.743 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-22T03:29:52.809 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-22T03:29:52.828 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-22T03:29:52.860 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-22T03:29:52.860 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-22T03:29:52.915 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-22T03:29:52.923 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-22T03:29:52.969 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-22T03:29:52.970 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-22T03:29:53.016 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-22T03:29:53.028 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-22T03:29:53.063 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-22T03:29:53.063 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-22T03:29:53.116 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-22T03:29:53.134 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-22T03:29:53.169 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-22T03:29:53.170 INFO:tasks.workunit.client.0.smithi070.stdout:2: acquiring exclusive lock 2024-09-22T03:29:53.221 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock acquired 2024-09-22T03:29:53.247 INFO:tasks.workunit.client.0.smithi070.stdout:2: releasing exclusive lock 2024-09-22T03:29:53.278 INFO:tasks.workunit.client.0.smithi070.stdout:2: exclusive lock released 2024-09-22T03:29:53.315 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-22T03:29:53.325 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-22T03:29:53.355 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-22T03:29:53.355 INFO:tasks.workunit.client.0.smithi070.stdout:1: acquiring exclusive lock 2024-09-22T03:29:53.393 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock acquired 2024-09-22T03:29:53.409 INFO:tasks.workunit.client.0.smithi070.stdout:1: releasing exclusive lock 2024-09-22T03:29:53.438 INFO:tasks.workunit.client.0.smithi070.stdout:1: exclusive lock released 2024-09-22T03:29:53.593 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ExclusiveLock (1733 ms) 2024-09-22T03:29:53.594 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.BreakLock 2024-09-22T03:29:53.594 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:11553: Skipped 2024-09-22T03:29:53.594 INFO:tasks.workunit.client.0.smithi070.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:29:53.594 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.BreakLock (0 ms) 2024-09-22T03:29:53.594 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-09-22T03:29:53.594 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:53.827 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (233 ms) 2024-09-22T03:29:53.827 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-09-22T03:29:53.827 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-09-22T03:29:53.827 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-09-22T03:29:53.827 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:54.402 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-22T03:29:54.402 INFO:tasks.workunit.client.0.smithi070.stdout:1.56% done 2024-09-22T03:29:54.402 INFO:tasks.workunit.client.0.smithi070.stdout:3.12% done 2024-09-22T03:29:54.402 INFO:tasks.workunit.client.0.smithi070.stdout:4.69% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:6.25% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:7.81% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:9.38% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:10.94% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:12.50% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:14.06% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:15.62% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:17.19% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:18.75% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:20.31% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:21.88% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:23.44% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-22T03:29:54.403 INFO:tasks.workunit.client.0.smithi070.stdout:26.56% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:28.12% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:29.69% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:31.25% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:32.81% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:34.38% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:35.94% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:37.50% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:39.06% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:40.62% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:42.19% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:43.75% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:45.31% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:46.88% done 2024-09-22T03:29:54.404 INFO:tasks.workunit.client.0.smithi070.stdout:48.44% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:51.56% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:53.12% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:54.69% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:56.25% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:57.81% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:59.38% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:60.94% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:62.50% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:64.06% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:65.62% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:67.19% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:68.75% done 2024-09-22T03:29:54.405 INFO:tasks.workunit.client.0.smithi070.stdout:70.31% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:71.88% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:73.44% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:76.56% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:78.12% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:79.69% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:81.25% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:82.81% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:84.38% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:85.94% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:87.50% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:89.06% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:90.62% done 2024-09-22T03:29:54.406 INFO:tasks.workunit.client.0.smithi070.stdout:92.19% done 2024-09-22T03:29:54.407 INFO:tasks.workunit.client.0.smithi070.stdout:93.75% done 2024-09-22T03:29:54.407 INFO:tasks.workunit.client.0.smithi070.stdout:95.31% done 2024-09-22T03:29:54.407 INFO:tasks.workunit.client.0.smithi070.stdout:96.88% done 2024-09-22T03:29:54.407 INFO:tasks.workunit.client.0.smithi070.stdout:98.44% done 2024-09-22T03:29:54.407 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurge (575 ms) 2024-09-22T03:29:54.407 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-09-22T03:29:54.407 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:55.041 INFO:tasks.workunit.client.0.smithi070.stdout:0.00% done 2024-09-22T03:29:55.041 INFO:tasks.workunit.client.0.smithi070.stdout:1.56% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:3.12% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:4.69% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:6.25% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:7.81% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:9.38% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:10.94% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:12.50% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:14.06% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:15.62% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:17.19% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:18.75% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:20.31% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:21.88% done 2024-09-22T03:29:55.042 INFO:tasks.workunit.client.0.smithi070.stdout:23.44% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:25.00% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:26.56% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:28.12% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:29.69% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:31.25% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:32.81% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:34.38% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:35.94% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:37.50% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:39.06% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:40.62% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:42.19% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:43.75% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:45.31% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:46.88% done 2024-09-22T03:29:55.043 INFO:tasks.workunit.client.0.smithi070.stdout:48.44% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:50.00% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:51.56% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:53.12% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:54.69% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:56.25% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:57.81% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:59.38% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:60.94% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:62.50% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:64.06% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:65.62% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:67.19% done 2024-09-22T03:29:55.044 INFO:tasks.workunit.client.0.smithi070.stdout:68.75% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:70.31% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:71.88% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:73.44% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:75.00% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:76.56% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:78.12% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:79.69% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:81.25% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:82.81% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:84.38% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:85.94% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:87.50% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:89.06% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:90.62% done 2024-09-22T03:29:55.045 INFO:tasks.workunit.client.0.smithi070.stdout:92.19% done 2024-09-22T03:29:55.046 INFO:tasks.workunit.client.0.smithi070.stdout:93.75% done 2024-09-22T03:29:55.046 INFO:tasks.workunit.client.0.smithi070.stdout:95.31% done 2024-09-22T03:29:55.046 INFO:tasks.workunit.client.0.smithi070.stdout:96.88% done 2024-09-22T03:29:55.046 INFO:tasks.workunit.client.0.smithi070.stdout:98.44% done 2024-09-22T03:29:55.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (640 ms) 2024-09-22T03:29:55.046 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-09-22T03:29:55.046 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:55.109 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:56.259 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestTrashPurge (1217 ms) 2024-09-22T03:29:56.259 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-09-22T03:29:56.259 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:56.675 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestTrashMoveAndRestore (416 ms) 2024-09-22T03:29:56.675 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-09-22T03:29:56.675 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:57.019 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestListWatchers (344 ms) 2024-09-22T03:29:57.019 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-09-22T03:29:57.019 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:57.936 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestSetSnapById (917 ms) 2024-09-22T03:29:57.936 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Namespaces 2024-09-22T03:29:58.144 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Namespaces (208 ms) 2024-09-22T03:29:58.144 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-09-22T03:29:58.377 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:58.397 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:59.139 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.NamespacesPP (995 ms) 2024-09-22T03:29:59.139 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Migration 2024-09-22T03:29:59.139 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:29:59.139 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:03.582 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Migration (4443 ms) 2024-09-22T03:30:03.582 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-09-22T03:30:03.582 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:03.582 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:07.805 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.MigrationPP (4223 ms) 2024-09-22T03:30:07.805 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-09-22T03:30:07.805 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:07.954 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetAccessTimestamp (149 ms) 2024-09-22T03:30:07.954 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-09-22T03:30:07.954 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:08.128 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.TestGetModifyTimestamp (175 ms) 2024-09-22T03:30:08.128 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-09-22T03:30:08.128 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:09.350 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ZeroOverlapFlatten (1221 ms) 2024-09-22T03:30:09.350 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-09-22T03:30:09.554 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.PoolMetadata (204 ms) 2024-09-22T03:30:09.554 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-09-22T03:30:09.701 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.PoolMetadataPP (148 ms) 2024-09-22T03:30:09.701 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.Config 2024-09-22T03:30:09.726 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:10.110 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.Config (408 ms) 2024-09-22T03:30:10.110 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-09-22T03:30:10.127 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:10.463 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ConfigPP (353 ms) 2024-09-22T03:30:10.463 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-09-22T03:30:12.448 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:14.701 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:14.782 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:14.870 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:16.130 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.PoolStatsPP (5668 ms) 2024-09-22T03:30:16.130 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-09-22T03:30:17.444 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:23.729 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.ImageSpec (7599 ms) 2024-09-22T03:30:23.729 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-09-22T03:30:25.447 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:36.760 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 3 2024-09-22T03:30:36.761 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 3 2024-09-22T03:30:36.761 INFO:tasks.workunit.client.0.smithi070.stdout:child: image223 2024-09-22T03:30:36.761 INFO:tasks.workunit.client.0.smithi070.stdout:child: image225 2024-09-22T03:30:36.761 INFO:tasks.workunit.client.0.smithi070.stdout:child: image224 2024-09-22T03:30:36.761 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-1/image223/106091a94adc 2024-09-22T03:30:36.761 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:30:36.761 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-20/image224/1060169de42c 2024-09-22T03:30:36.762 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:30:36.762 INFO:tasks.workunit.client.0.smithi070.stdout:found child test-librbd-smithi070-29639-1/image225/1060a6e90c5e 2024-09-22T03:30:36.762 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:30:36.762 INFO:tasks.workunit.client.0.smithi070.stdout:num children is: 0 2024-09-22T03:30:36.762 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:30:36.762 INFO:tasks.workunit.client.0.smithi070.stdout:num snaps is: 0 2024-09-22T03:30:36.763 INFO:tasks.workunit.client.0.smithi070.stdout:expected: 0 2024-09-22T03:30:36.763 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.SnapRemoveWithChildMissing (13031 ms) 2024-09-22T03:30:36.763 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-09-22T03:30:38.482 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:47.437 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.QuiesceWatch (10677 ms) 2024-09-22T03:30:47.438 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-09-22T03:30:47.438 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:54.430 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (6992 ms) 2024-09-22T03:30:54.430 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-09-22T03:30:54.430 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:13115: Skipped 2024-09-22T03:30:54.430 INFO:tasks.workunit.client.0.smithi070.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:30:54.430 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError (0 ms) 2024-09-22T03:30:54.430 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-09-22T03:30:54.430 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:30:54.557 INFO:tasks.workunit.client.0.smithi070.stderr:test quiesce is not long enough to time out 2024-09-22T03:31:03.820 INFO:tasks.workunit.client.0.smithi070.stderr:test quiesce is timed out 2024-09-22T03:31:03.856 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 0 2024-09-22T03:31:04.856 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 1 2024-09-22T03:31:05.856 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 2 2024-09-22T03:31:06.856 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 3 2024-09-22T03:31:07.857 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 4 2024-09-22T03:31:08.857 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 5 2024-09-22T03:31:09.857 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 6 2024-09-22T03:31:10.857 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 7 2024-09-22T03:31:11.857 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 8 2024-09-22T03:31:12.857 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 9 2024-09-22T03:31:13.857 INFO:tasks.workunit.client.0.smithi070.stderr:waiting for timed out ... 10 2024-09-22T03:31:24.858 INFO:tasks.workunit.client.0.smithi070.stderr:test retry succeeds 2024-09-22T03:31:28.451 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (34021 ms) 2024-09-22T03:31:28.451 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-09-22T03:31:28.451 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:31:28.788 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.WriteZeroes (337 ms) 2024-09-22T03:31:28.788 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-09-22T03:31:28.788 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:31:29.175 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (387 ms) 2024-09-22T03:31:29.304 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestLibRBD.ConcurrentOperations 2024-09-22T03:31:29.305 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_librbd.cc:13474: Skipped 2024-09-22T03:31:29.305 INFO:tasks.workunit.client.0.smithi070.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:31:29.305 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations (0 ms) 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 170 tests from TestLibRBD (872285 ms total) 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 170 tests from 1 test suite ran. (891772 ms total) 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 163 tests. 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] 7 tests, listed below: 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.BreakLock 2024-09-22T03:31:48.624 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError 2024-09-22T03:31:48.625 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations 2024-09-22T03:31:48.625 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:31:48.625 INFO:tasks.workunit.client.0.smithi070.stdout: YOU HAVE 1 DISABLED TEST 2024-09-22T03:31:48.625 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:31:48.716 INFO:tasks.workunit.client.0.smithi070.stdout:seed 38714 2024-09-22T03:31:48.717 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = EncryptedFlattenTest/0.* 2024-09-22T03:31:48.717 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 7 tests from 1 test suite. 2024-09-22T03:31:48.717 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:31:48.717 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/0, where TypeParam = PlaintextUnderLUKS1 2024-09-22T03:31:48.765 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.Simple 2024-09-22T03:31:50.626 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:32:03.925 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.Simple (15161 ms) 2024-09-22T03:32:03.925 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.Grow 2024-09-22T03:32:03.925 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:32:14.475 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.Grow (10550 ms) 2024-09-22T03:32:14.476 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.Shrink 2024-09-22T03:32:14.476 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:32:25.897 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.Shrink (11422 ms) 2024-09-22T03:32:25.898 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.ShrinkToOne 2024-09-22T03:32:25.898 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:32:35.960 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.ShrinkToOne (10062 ms) 2024-09-22T03:32:35.960 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.ShrinkToOneAfterSnapshot 2024-09-22T03:32:35.960 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:32:49.425 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.ShrinkToOneAfterSnapshot (13466 ms) 2024-09-22T03:32:49.425 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.MinOverlap 2024-09-22T03:32:49.425 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:32:59.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.MinOverlap (10522 ms) 2024-09-22T03:32:59.947 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/0.ZeroOverlap 2024-09-22T03:32:59.948 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:33:10.383 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/0.ZeroOverlap (10435 ms) 2024-09-22T03:33:10.497 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/0 (81618 ms total) 2024-09-22T03:33:10.497 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:33:10.497 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:33:10.497 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 7 tests from 1 test suite ran. (81781 ms total) 2024-09-22T03:33:10.497 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 7 tests. 2024-09-22T03:33:10.569 INFO:tasks.workunit.client.0.smithi070.stdout:seed 38783 2024-09-22T03:33:10.569 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = EncryptedFlattenTest/1.* 2024-09-22T03:33:10.569 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 7 tests from 1 test suite. 2024-09-22T03:33:10.570 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:33:10.570 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/1, where TypeParam = PlaintextUnderLUKS2 2024-09-22T03:33:10.640 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.Simple 2024-09-22T03:33:12.472 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:33:29.855 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.Simple (19215 ms) 2024-09-22T03:33:29.855 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.Grow 2024-09-22T03:33:29.855 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:33:44.915 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.Grow (15061 ms) 2024-09-22T03:33:44.915 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.Shrink 2024-09-22T03:33:44.915 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:34:01.235 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.Shrink (16319 ms) 2024-09-22T03:34:01.235 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.ShrinkToOne 2024-09-22T03:34:01.235 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:34:16.822 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.ShrinkToOne (15588 ms) 2024-09-22T03:34:16.823 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.ShrinkToOneAfterSnapshot 2024-09-22T03:34:16.823 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:34:36.351 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.ShrinkToOneAfterSnapshot (19529 ms) 2024-09-22T03:34:36.351 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.MinOverlap 2024-09-22T03:34:36.351 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:34:49.814 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.MinOverlap (13463 ms) 2024-09-22T03:34:49.814 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/1.ZeroOverlap 2024-09-22T03:34:49.815 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:35:03.385 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/1.ZeroOverlap (13570 ms) 2024-09-22T03:35:03.493 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/1 (112745 ms total) 2024-09-22T03:35:03.493 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:35:03.493 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:35:03.493 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 7 tests from 1 test suite ran. (112923 ms total) 2024-09-22T03:35:03.493 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 7 tests. 2024-09-22T03:35:03.565 INFO:tasks.workunit.client.0.smithi070.stdout:seed 44803 2024-09-22T03:35:03.565 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = EncryptedFlattenTest/2.* 2024-09-22T03:35:03.565 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 7 tests from 1 test suite. 2024-09-22T03:35:03.565 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:35:03.566 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/2, where TypeParam = UnformattedLUKS1 2024-09-22T03:35:03.613 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.Simple 2024-09-22T03:35:05.499 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:35:20.700 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.Simple (17087 ms) 2024-09-22T03:35:20.700 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.Grow 2024-09-22T03:35:20.701 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:35:33.815 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.Grow (13114 ms) 2024-09-22T03:35:33.815 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.Shrink 2024-09-22T03:35:33.815 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:35:47.385 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.Shrink (13570 ms) 2024-09-22T03:35:47.385 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.ShrinkToOne 2024-09-22T03:35:47.385 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:36:00.188 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.ShrinkToOne (12803 ms) 2024-09-22T03:36:00.188 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.ShrinkToOneAfterSnapshot 2024-09-22T03:36:00.188 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:36:16.509 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.ShrinkToOneAfterSnapshot (16322 ms) 2024-09-22T03:36:16.510 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.MinOverlap 2024-09-22T03:36:16.510 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:36:29.739 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.MinOverlap (13229 ms) 2024-09-22T03:36:29.739 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/2.ZeroOverlap 2024-09-22T03:36:29.739 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:36:42.004 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/2.ZeroOverlap (12265 ms) 2024-09-22T03:36:42.536 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/2 (98390 ms total) 2024-09-22T03:36:42.536 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:36:42.536 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:36:42.536 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 7 tests from 1 test suite ran. (98970 ms total) 2024-09-22T03:36:42.536 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 7 tests. 2024-09-22T03:36:42.619 INFO:tasks.workunit.client.0.smithi070.stdout:seed 44872 2024-09-22T03:36:42.620 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = EncryptedFlattenTest/3.* 2024-09-22T03:36:42.620 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 7 tests from 1 test suite. 2024-09-22T03:36:42.620 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:36:42.620 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/3, where TypeParam = LUKS1UnderLUKS1 2024-09-22T03:36:42.678 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.Simple 2024-09-22T03:36:44.545 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:37:03.871 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.Simple (21193 ms) 2024-09-22T03:37:03.871 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.Grow 2024-09-22T03:37:03.872 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:37:20.738 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.Grow (16866 ms) 2024-09-22T03:37:20.738 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.Shrink 2024-09-22T03:37:20.738 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:37:38.121 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.Shrink (17384 ms) 2024-09-22T03:37:38.121 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.ShrinkToOne 2024-09-22T03:37:38.121 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:37:54.024 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.ShrinkToOne (15903 ms) 2024-09-22T03:37:54.024 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.ShrinkToOneAfterSnapshot 2024-09-22T03:37:54.025 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:38:12.198 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.ShrinkToOneAfterSnapshot (18174 ms) 2024-09-22T03:38:12.198 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.MinOverlap 2024-09-22T03:38:12.199 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:38:28.042 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.MinOverlap (15844 ms) 2024-09-22T03:38:28.043 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/3.ZeroOverlap 2024-09-22T03:38:28.043 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:38:43.497 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/3.ZeroOverlap (15455 ms) 2024-09-22T03:38:43.525 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/3 (120820 ms total) 2024-09-22T03:38:43.525 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:38:43.526 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:38:43.526 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 7 tests from 1 test suite ran. (120905 ms total) 2024-09-22T03:38:43.526 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 7 tests. 2024-09-22T03:38:43.598 INFO:tasks.workunit.client.0.smithi070.stdout:seed 44944 2024-09-22T03:38:43.599 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = EncryptedFlattenTest/4.* 2024-09-22T03:38:43.599 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 7 tests from 1 test suite. 2024-09-22T03:38:43.599 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:38:43.599 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/4, where TypeParam = LUKS1UnderLUKS2 2024-09-22T03:38:43.667 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.Simple 2024-09-22T03:38:45.535 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:39:09.617 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.Simple (25950 ms) 2024-09-22T03:39:09.617 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.Grow 2024-09-22T03:39:09.617 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:39:28.662 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.Grow (19044 ms) 2024-09-22T03:39:28.662 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.Shrink 2024-09-22T03:39:28.662 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:39:47.657 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.Shrink (18996 ms) 2024-09-22T03:39:47.657 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.ShrinkToOne 2024-09-22T03:39:47.657 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:40:08.503 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.ShrinkToOne (20846 ms) 2024-09-22T03:40:08.504 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.ShrinkToOneAfterSnapshot 2024-09-22T03:40:08.504 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:40:37.029 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.ShrinkToOneAfterSnapshot (28525 ms) 2024-09-22T03:40:37.029 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.MinOverlap 2024-09-22T03:40:37.029 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:40:57.846 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.MinOverlap (20817 ms) 2024-09-22T03:40:57.846 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/4.ZeroOverlap 2024-09-22T03:40:57.846 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:41:16.186 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/4.ZeroOverlap (18340 ms) 2024-09-22T03:41:16.618 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/4 (152518 ms total) 2024-09-22T03:41:16.618 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:41:16.618 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:41:16.618 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 7 tests from 1 test suite ran. (153018 ms total) 2024-09-22T03:41:16.618 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 7 tests. 2024-09-22T03:41:16.670 INFO:tasks.workunit.client.0.smithi070.stdout:seed 50808 2024-09-22T03:41:16.670 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = EncryptedFlattenTest/5.* 2024-09-22T03:41:16.670 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 7 tests from 1 test suite. 2024-09-22T03:41:16.670 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:41:16.671 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/5, where TypeParam = UnformattedLUKS2 2024-09-22T03:41:16.740 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.Simple 2024-09-22T03:41:18.627 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:41:34.372 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.Simple (17631 ms) 2024-09-22T03:41:34.372 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.Grow 2024-09-22T03:41:34.372 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:41:54.700 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.Grow (20328 ms) 2024-09-22T03:41:54.700 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.Shrink 2024-09-22T03:41:54.700 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:42:13.490 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.Shrink (18790 ms) 2024-09-22T03:42:13.490 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.ShrinkToOne 2024-09-22T03:42:13.490 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:42:27.765 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.ShrinkToOne (14275 ms) 2024-09-22T03:42:27.765 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.ShrinkToOneAfterSnapshot 2024-09-22T03:42:27.765 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:42:46.883 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.ShrinkToOneAfterSnapshot (19118 ms) 2024-09-22T03:42:46.883 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.MinOverlap 2024-09-22T03:42:46.883 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:43:04.176 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.MinOverlap (17294 ms) 2024-09-22T03:43:04.176 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/5.ZeroOverlap 2024-09-22T03:43:04.176 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:43:20.484 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/5.ZeroOverlap (16307 ms) 2024-09-22T03:43:20.744 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/5 (123743 ms total) 2024-09-22T03:43:20.745 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:43:20.745 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:43:20.745 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 7 tests from 1 test suite ran. (124074 ms total) 2024-09-22T03:43:20.745 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 7 tests. 2024-09-22T03:43:20.819 INFO:tasks.workunit.client.0.smithi070.stdout:seed 56510 2024-09-22T03:43:20.820 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = EncryptedFlattenTest/6.* 2024-09-22T03:43:20.820 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 7 tests from 1 test suite. 2024-09-22T03:43:20.820 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:43:20.820 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/6, where TypeParam = LUKS2UnderLUKS2 2024-09-22T03:43:20.872 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.Simple 2024-09-22T03:43:22.560 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:43:59.038 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.Simple (38166 ms) 2024-09-22T03:43:59.038 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.Grow 2024-09-22T03:43:59.038 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:44:25.030 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.Grow (25992 ms) 2024-09-22T03:44:25.030 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.Shrink 2024-09-22T03:44:25.030 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:44:48.327 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.Shrink (23298 ms) 2024-09-22T03:44:48.327 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.ShrinkToOne 2024-09-22T03:44:48.327 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:45:11.124 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.ShrinkToOne (22796 ms) 2024-09-22T03:45:11.124 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.ShrinkToOneAfterSnapshot 2024-09-22T03:45:11.124 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:45:45.185 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.ShrinkToOneAfterSnapshot (34062 ms) 2024-09-22T03:45:45.185 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.MinOverlap 2024-09-22T03:45:45.186 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:46:04.234 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.MinOverlap (19049 ms) 2024-09-22T03:46:04.234 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/6.ZeroOverlap 2024-09-22T03:46:04.234 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:46:29.946 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/6.ZeroOverlap (25712 ms) 2024-09-22T03:46:30.639 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/6 (189075 ms total) 2024-09-22T03:46:30.639 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:46:30.639 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:46:30.639 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 7 tests from 1 test suite ran. (189820 ms total) 2024-09-22T03:46:30.639 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 7 tests. 2024-09-22T03:46:30.689 INFO:tasks.workunit.client.0.smithi070.stdout:seed 67653 2024-09-22T03:46:30.689 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = EncryptedFlattenTest/7.* 2024-09-22T03:46:30.689 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 7 tests from 1 test suite. 2024-09-22T03:46:30.689 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:46:30.689 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/7, where TypeParam = LUKS2UnderLUKS1 2024-09-22T03:46:30.734 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.Simple 2024-09-22T03:46:32.646 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:46:59.867 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.Simple (29132 ms) 2024-09-22T03:46:59.867 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.Grow 2024-09-22T03:46:59.867 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:47:24.875 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.Grow (25008 ms) 2024-09-22T03:47:24.875 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.Shrink 2024-09-22T03:47:24.875 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:47:51.135 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.Shrink (26260 ms) 2024-09-22T03:47:51.135 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.ShrinkToOne 2024-09-22T03:47:51.135 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:48:13.237 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.ShrinkToOne (22103 ms) 2024-09-22T03:48:13.238 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.ShrinkToOneAfterSnapshot 2024-09-22T03:48:13.238 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:48:40.247 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.ShrinkToOneAfterSnapshot (27010 ms) 2024-09-22T03:48:40.248 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.MinOverlap 2024-09-22T03:48:40.248 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:49:07.651 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.MinOverlap (27404 ms) 2024-09-22T03:49:07.651 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] EncryptedFlattenTest/7.ZeroOverlap 2024-09-22T03:49:07.651 INFO:tasks.workunit.client.0.smithi070.stdout:using new format! 2024-09-22T03:49:31.883 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] EncryptedFlattenTest/7.ZeroOverlap (24231 ms) 2024-09-22T03:49:32.659 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 7 tests from EncryptedFlattenTest/7 (181148 ms total) 2024-09-22T03:49:32.659 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:49:32.659 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:49:32.659 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 7 tests from 1 test suite ran. (181970 ms total) 2024-09-22T03:49:32.659 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 7 tests. 2024-09-22T03:49:32.721 INFO:tasks.workunit.client.0.smithi070.stdout:seed 73709 2024-09-22T03:49:32.722 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = TestImageWatcher.* 2024-09-22T03:49:32.722 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 17 tests from 1 test suite. 2024-09-22T03:49:32.722 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:49:32.722 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 17 tests from TestImageWatcher 2024-09-22T03:49:34.663 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyHeaderUpdate 2024-09-22T03:49:38.210 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyHeaderUpdate (3546 ms) 2024-09-22T03:49:38.210 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyFlatten 2024-09-22T03:49:38.782 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyFlatten (573 ms) 2024-09-22T03:49:38.782 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyResize 2024-09-22T03:49:40.606 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyResize (1824 ms) 2024-09-22T03:49:40.606 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyRebuildObjectMap 2024-09-22T03:49:40.980 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyRebuildObjectMap (374 ms) 2024-09-22T03:49:40.980 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyUpdateFeatures 2024-09-22T03:49:41.286 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyUpdateFeatures (306 ms) 2024-09-22T03:49:41.286 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapCreate 2024-09-22T03:49:41.611 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapCreate (324 ms) 2024-09-22T03:49:41.611 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapCreateError 2024-09-22T03:49:41.902 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapCreateError (291 ms) 2024-09-22T03:49:41.902 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapRename 2024-09-22T03:49:42.196 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapRename (295 ms) 2024-09-22T03:49:42.196 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapRenameError 2024-09-22T03:49:42.472 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapRenameError (275 ms) 2024-09-22T03:49:42.472 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapRemove 2024-09-22T03:49:42.797 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapRemove (326 ms) 2024-09-22T03:49:42.797 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapProtect 2024-09-22T03:49:43.140 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapProtect (343 ms) 2024-09-22T03:49:43.140 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifySnapUnprotect 2024-09-22T03:49:43.499 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifySnapUnprotect (358 ms) 2024-09-22T03:49:43.499 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyRename 2024-09-22T03:49:43.736 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyRename (238 ms) 2024-09-22T03:49:43.736 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyAsyncTimedOut 2024-09-22T03:49:43.962 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyAsyncTimedOut (225 ms) 2024-09-22T03:49:43.962 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyAsyncError 2024-09-22T03:49:44.162 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyAsyncError (201 ms) 2024-09-22T03:49:44.163 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyAsyncCompleteError 2024-09-22T03:49:44.384 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyAsyncCompleteError (221 ms) 2024-09-22T03:49:44.384 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestImageWatcher.NotifyAsyncRequestTimedOut 2024-09-22T03:49:44.571 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestImageWatcher.NotifyAsyncRequestTimedOut (188 ms) 2024-09-22T03:49:44.622 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 17 tests from TestImageWatcher (9908 ms total) 2024-09-22T03:49:44.622 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:49:44.622 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:49:44.622 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 17 tests from 1 test suite ran. (11901 ms total) 2024-09-22T03:49:44.622 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 17 tests. 2024-09-22T03:49:44.671 INFO:tasks.workunit.client.0.smithi070.stdout:seed 73775 2024-09-22T03:49:44.672 INFO:tasks.workunit.client.0.smithi070.stdout:Note: Google Test filter = TestInternal.* 2024-09-22T03:49:44.672 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] Running 34 tests from 1 test suite. 2024-09-22T03:49:44.672 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment set-up. 2024-09-22T03:49:44.672 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 34 tests from TestInternal 2024-09-22T03:49:46.628 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.OpenByID 2024-09-22T03:49:48.890 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.OpenByID (2262 ms) 2024-09-22T03:49:48.891 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.OpenSnapDNE 2024-09-22T03:49:49.090 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.OpenSnapDNE (200 ms) 2024-09-22T03:49:49.090 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.IsExclusiveLockOwner 2024-09-22T03:49:49.285 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.IsExclusiveLockOwner (194 ms) 2024-09-22T03:49:49.285 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.ResizeLocksImage 2024-09-22T03:49:49.528 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.ResizeLocksImage (243 ms) 2024-09-22T03:49:49.528 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.ResizeFailsToLockImage 2024-09-22T03:49:49.703 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.ResizeFailsToLockImage (176 ms) 2024-09-22T03:49:49.703 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapCreateLocksImage 2024-09-22T03:49:51.792 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapCreateLocksImage (2088 ms) 2024-09-22T03:49:51.792 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapCreateFailsToLockImage 2024-09-22T03:49:51.948 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapCreateFailsToLockImage (156 ms) 2024-09-22T03:49:51.948 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapRollbackLocksImage 2024-09-22T03:49:53.806 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapRollbackLocksImage (1859 ms) 2024-09-22T03:49:53.806 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapRollbackFailsToLockImage 2024-09-22T03:49:55.823 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapRollbackFailsToLockImage (2016 ms) 2024-09-22T03:49:55.823 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapSetReleasesLock 2024-09-22T03:49:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapSetReleasesLock (1989 ms) 2024-09-22T03:49:57.811 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.FlattenLocksImage 2024-09-22T03:49:59.860 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.FlattenLocksImage (2048 ms) 2024-09-22T03:49:59.860 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.FlattenFailsToLockImage 2024-09-22T03:50:02.855 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.FlattenFailsToLockImage (2996 ms) 2024-09-22T03:50:02.855 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklisted 2024-09-22T03:50:02.936 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_internal.cc:382: Skipped 2024-09-22T03:50:02.937 INFO:tasks.workunit.client.0.smithi070.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:50:02.937 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.WriteFailsToLockImageBlocklisted (81 ms) 2024-09-22T03:50:02.937 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklistedWatch 2024-09-22T03:50:03.004 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_internal.cc:416: Skipped 2024-09-22T03:50:03.004 INFO:tasks.workunit.client.0.smithi070.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:50:03.004 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.WriteFailsToLockImageBlocklistedWatch (67 ms) 2024-09-22T03:50:03.004 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.AioWriteRequestsLock 2024-09-22T03:50:03.277 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.AioWriteRequestsLock (273 ms) 2024-09-22T03:50:03.277 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.AioDiscardRequestsLock 2024-09-22T03:50:03.491 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.AioDiscardRequestsLock (215 ms) 2024-09-22T03:50:03.491 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.CancelAsyncResize 2024-09-22T03:50:03.925 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.CancelAsyncResize (434 ms) 2024-09-22T03:50:03.925 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.MultipleResize 2024-09-22T03:50:04.571 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.MultipleResize (646 ms) 2024-09-22T03:50:04.571 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.Metadata 2024-09-22T03:50:05.078 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.Metadata (507 ms) 2024-09-22T03:50:05.079 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.MetadataConfApply 2024-09-22T03:50:05.373 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.MetadataConfApply (294 ms) 2024-09-22T03:50:05.373 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapshotCopyup 2024-09-22T03:50:05.439 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_internal.cc:662: Skipped 2024-09-22T03:50:05.439 INFO:tasks.workunit.client.0.smithi070.stdout:Not supported by crimson yet. Skipped 2024-09-22T03:50:05.439 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.SnapshotCopyup (66 ms) 2024-09-22T03:50:05.439 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapshotCopyupZeros 2024-09-22T03:50:06.977 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapshotCopyupZeros (1539 ms) 2024-09-22T03:50:06.978 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SnapshotCopyupZerosMigration 2024-09-22T03:50:08.957 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SnapshotCopyupZerosMigration (1979 ms) 2024-09-22T03:50:08.957 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.ResizeCopyup 2024-09-22T03:50:11.334 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.ResizeCopyup (2377 ms) 2024-09-22T03:50:11.334 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.DiscardCopyup 2024-09-22T03:50:11.402 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_internal.cc:1056: Skipped 2024-09-22T03:50:11.402 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to unmet REQUIRE 2024-09-22T03:50:11.402 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.DiscardCopyup (68 ms) 2024-09-22T03:50:11.402 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.ImageOptions 2024-09-22T03:50:11.472 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.ImageOptions (71 ms) 2024-09-22T03:50:11.473 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.WriteFullCopyup 2024-09-22T03:50:15.858 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.WriteFullCopyup (4386 ms) 2024-09-22T03:50:15.859 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.DiffIterateCloneOverwrite 2024-09-22T03:50:17.994 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.DiffIterateCloneOverwrite (2136 ms) 2024-09-22T03:50:17.995 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.TestCoR 2024-09-22T03:50:18.067 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_internal.cc:1313: Skipped 2024-09-22T03:50:18.067 INFO:tasks.workunit.client.0.smithi070.stdout:Skipping due to disabled copy-on-read 2024-09-22T03:50:18.067 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.TestCoR (73 ms) 2024-09-22T03:50:18.068 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.FlattenNoEmptyObjects 2024-09-22T03:50:21.302 INFO:tasks.workunit.client.0.smithi070.stdout:made parent image "image39": 4194304K (1024 * 4K) 2024-09-22T03:50:21.302 INFO:tasks.workunit.client.0.smithi070.stdout:generated random write map: 2024-09-22T03:50:21.302 INFO:tasks.workunit.client.0.smithi070.stdout: [49 , 201847 ] 2024-09-22T03:50:21.302 INFO:tasks.workunit.client.0.smithi070.stdout: [65 , 267217 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [73 , 301084 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [78 , 322870 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [105 , 430880 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [131 , 536633 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [133 , 545412 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [137 , 562242 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [144 , 593285 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [147 , 605452 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [164 , 674633 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [169 , 693669 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [180 , 739071 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [196 , 805642 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [206 , 845312 ] 2024-09-22T03:50:21.303 INFO:tasks.workunit.client.0.smithi070.stdout: [227 , 932437 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [232 , 951505 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [234 , 961080 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [265 , 1086200 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [269 , 1102579 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [280 , 1148798 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [296 , 1214957 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [297 , 1217793 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [298 , 1220622 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [304 , 1247281 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [314 , 1287571 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [321 , 1315244 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [331 , 1356391 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [334 , 1369026 ] 2024-09-22T03:50:21.304 INFO:tasks.workunit.client.0.smithi070.stdout: [338 , 1385091 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [346 , 1420236 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [357 , 1464278 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [366 , 1499443 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [372 , 1525691 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [379 , 1555608 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [386 , 1584625 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [394 , 1615948 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [406 , 1663721 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [409 , 1678621 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [410 , 1680454 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [413 , 1693555 ] 2024-09-22T03:50:21.305 INFO:tasks.workunit.client.0.smithi070.stdout: [428 , 1756595 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [438 , 1794266 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [442 , 1813564 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [461 , 1891419 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [471 , 1931005 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [512 , 2097338 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [519 , 2127799 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [522 , 2140110 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [529 , 2170166 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [534 , 2188947 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [540 , 2213683 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [550 , 2254476 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [554 , 2271762 ] 2024-09-22T03:50:21.306 INFO:tasks.workunit.client.0.smithi070.stdout: [564 , 2312728 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [576 , 2360638 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [607 , 2487028 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [621 , 2546845 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [630 , 2581457 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [652 , 2673471 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [654 , 2682072 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [659 , 2700863 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [660 , 2704033 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [678 , 2779324 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [693 , 2838803 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [699 , 2864867 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [702 , 2876728 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [709 , 2904469 ] 2024-09-22T03:50:21.307 INFO:tasks.workunit.client.0.smithi070.stdout: [747 , 3060514 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [749 , 3071050 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [757 , 3102038 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [760 , 3116078 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [764 , 3131338 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [773 , 3169076 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [808 , 3311349 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [811 , 3322702 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [812 , 3328132 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [828 , 3393301 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [837 , 3431696 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [843 , 3455728 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [850 , 3483828 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [856 , 3507255 ] 2024-09-22T03:50:21.308 INFO:tasks.workunit.client.0.smithi070.stdout: [878 , 3599661 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [887 , 3635314 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [889 , 3641790 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [900 , 3688849 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [904 , 3703734 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [930 , 3810809 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [934 , 3828704 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [944 , 3866691 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [974 , 3991478 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [1006 , 4122773 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [1008 , 4130349 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: [1011 , 4142805 ] 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout:write data based on random map 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: write object-49 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: write object-65 2024-09-22T03:50:21.309 INFO:tasks.workunit.client.0.smithi070.stdout: write object-73 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-78 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-105 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-131 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-133 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-137 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-144 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-147 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-164 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-169 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-180 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-196 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-206 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-227 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-232 2024-09-22T03:50:21.310 INFO:tasks.workunit.client.0.smithi070.stdout: write object-234 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-265 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-269 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-280 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-296 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-297 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-298 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-304 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-314 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-321 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-331 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-334 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-338 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-346 2024-09-22T03:50:21.311 INFO:tasks.workunit.client.0.smithi070.stdout: write object-357 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-366 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-372 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-379 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-386 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-394 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-406 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-409 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-410 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-413 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-428 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-438 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-442 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-461 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-471 2024-09-22T03:50:21.312 INFO:tasks.workunit.client.0.smithi070.stdout: write object-512 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-519 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-522 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-529 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-534 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-540 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-550 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-554 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-564 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-576 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-607 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-621 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-630 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-652 2024-09-22T03:50:21.313 INFO:tasks.workunit.client.0.smithi070.stdout: write object-654 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-659 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-660 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-678 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-693 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-699 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-702 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-709 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-747 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-749 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-757 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-760 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-764 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-773 2024-09-22T03:50:21.314 INFO:tasks.workunit.client.0.smithi070.stdout: write object-808 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-811 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-812 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-828 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-837 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-843 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-850 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-856 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-878 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-887 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-889 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-900 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-904 2024-09-22T03:50:21.315 INFO:tasks.workunit.client.0.smithi070.stdout: write object-930 2024-09-22T03:50:21.316 INFO:tasks.workunit.client.0.smithi070.stdout: write object-934 2024-09-22T03:50:21.316 INFO:tasks.workunit.client.0.smithi070.stdout: write object-944 2024-09-22T03:50:21.316 INFO:tasks.workunit.client.0.smithi070.stdout: write object-974 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout: write object-1006 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout: write object-1008 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout: write object-1011 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout:verify written data by reading 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout: read object-49 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout:made snapshot "image39@parent_snap" and protect it 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout:made and opened clone "image40" 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout:flattening clone: "image40" 2024-09-22T03:50:24.937 INFO:tasks.workunit.client.0.smithi070.stdout:check whether child image has the same set of objects as parent 2024-09-22T03:50:24.938 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.FlattenNoEmptyObjects (6869 ms) 2024-09-22T03:50:24.938 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.PoolMetadataConfApply 2024-09-22T03:50:25.578 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.PoolMetadataConfApply (641 ms) 2024-09-22T03:50:25.578 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.Sparsify 2024-09-22T03:50:25.832 INFO:tasks.workunit.client.0.smithi070.stdout:sparsify_supported=1 2024-09-22T03:50:25.832 INFO:tasks.workunit.client.0.smithi070.stdout:sparse_read_supported=0 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_internal.cc:1730: Failure 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout:Expected equality of these values: 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout: static_cast(expected_m.size()) 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout: Which is: 1 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout: ictx->data_ctx.sparse_read(oid, m, read_bl, read_len, 0) 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout: Which is: 2 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_internal.cc:1732: Failure 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout:Expected equality of these values: 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout: m 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout: Which is: { (4096, 4096), (12288, 4096) } 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout: expected_m 2024-09-22T03:50:26.264 INFO:tasks.workunit.client.0.smithi070.stdout: Which is: { (0, 16384) } 2024-09-22T03:50:26.265 INFO:tasks.workunit.client.0.smithi070.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-373-g4e45e6b4/rpm/el9/BUILD/ceph-19.1.1-373-g4e45e6b4/src/test/librbd/test_internal.cc:1733: Failure 2024-09-22T03:50:26.265 INFO:tasks.workunit.client.0.smithi070.stdout:Value of: bl.contents_equal(read_bl) 2024-09-22T03:50:26.265 INFO:tasks.workunit.client.0.smithi070.stdout: Actual: false 2024-09-22T03:50:26.265 INFO:tasks.workunit.client.0.smithi070.stdout:Expected: true 2024-09-22T03:50:26.310 INFO:tasks.workunit.client.0.smithi070.stdout:[ FAILED ] TestInternal.Sparsify (733 ms) 2024-09-22T03:50:26.310 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.SparsifyClone 2024-09-22T03:50:26.483 INFO:tasks.workunit.client.0.smithi070.stdout:sparsify_supported=1 2024-09-22T03:50:28.862 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.SparsifyClone (2551 ms) 2024-09-22T03:50:28.862 INFO:tasks.workunit.client.0.smithi070.stdout:[ RUN ] TestInternal.MissingDataPool 2024-09-22T03:50:30.753 INFO:tasks.workunit.client.0.smithi070.stdout:[ OK ] TestInternal.MissingDataPool (1892 ms) 2024-09-22T03:50:30.775 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] 34 tests from TestInternal (44125 ms total) 2024-09-22T03:50:30.775 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:50:30.775 INFO:tasks.workunit.client.0.smithi070.stdout:[----------] Global test environment tear-down 2024-09-22T03:50:30.775 INFO:tasks.workunit.client.0.smithi070.stdout:[==========] 34 tests from 1 test suite ran. (46104 ms total) 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ PASSED ] 28 tests. 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] 5 tests, listed below: 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.WriteFailsToLockImageBlocklisted 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.WriteFailsToLockImageBlocklistedWatch 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.SnapshotCopyup 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.DiscardCopyup 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ SKIPPED ] TestInternal.TestCoR 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ FAILED ] 1 test, listed below: 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout:[ FAILED ] TestInternal.Sparsify 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout: 2024-09-22T03:50:30.776 INFO:tasks.workunit.client.0.smithi070.stdout: 1 FAILED TEST 2024-09-22T03:50:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:50:30.783 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2024-09-22T03:50:30.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-22T03:50:31.119 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-09-22T03:50:31.291 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=58d1e69174f84835a4be824dc48d16ea Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-09-22T03:50:31.294 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T03:50:31.303 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:50:31.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:50:31.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:31.437+0000 7fcb812d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:31.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:31.446+0000 7fcb812d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:31.638 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:50:31.638 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-22T03:50:31.656 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1488,"stamp":"2024-09-22T03:50:30.768917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":72316,"kb_used_data":72316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156820868,"statfs":{"total":160658620416,"available":160584568832,"internally_reserved":0,"allocated":74051584,"data_stored":74051584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4599727,"num_objects":-291,"num_object_clones":-3,"num_object_copies":-582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":-4931,"num_read_kb":-13685,"num_write":-929,"num_write_kb":-13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011033"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1324,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.779601+0000","last_change":"2024-09-22T03:15:55.475462+0000","last_active":"2024-09-22T03:50:26.779601+0000","last_peered":"2024-09-22T03:50:26.779601+0000","last_clean":"2024-09-22T03:50:26.779601+0000","last_became_active":"2024-09-22T03:15:53.559125+0000","last_became_peered":"2024-09-22T03:15:53.559125+0000","last_unstale":"2024-09-22T03:50:26.779601+0000","last_undegraded":"2024-09-22T03:50:26.779601+0000","last_fullsized":"2024-09-22T03:50:26.779601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1324,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.778916+0000","last_change":"2024-09-22T03:15:55.476029+0000","last_active":"2024-09-22T03:50:26.778916+0000","last_peered":"2024-09-22T03:50:26.778916+0000","last_clean":"2024-09-22T03:50:26.778916+0000","last_became_active":"2024-09-22T03:15:53.591886+0000","last_became_peered":"2024-09-22T03:15:53.591886+0000","last_unstale":"2024-09-22T03:50:26.778916+0000","last_undegraded":"2024-09-22T03:50:26.778916+0000","last_fullsized":"2024-09-22T03:50:26.778916+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1324,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.777102+0000","last_change":"2024-09-22T03:15:55.475380+0000","last_active":"2024-09-22T03:50:26.777102+0000","last_peered":"2024-09-22T03:50:26.777102+0000","last_clean":"2024-09-22T03:50:26.777102+0000","last_became_active":"2024-09-22T03:15:53.569486+0000","last_became_peered":"2024-09-22T03:15:53.569486+0000","last_unstale":"2024-09-22T03:50:26.777102+0000","last_undegraded":"2024-09-22T03:50:26.777102+0000","last_fullsized":"2024-09-22T03:50:26.777102+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1324,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.781903+0000","last_change":"2024-09-22T03:15:55.475811+0000","last_active":"2024-09-22T03:50:26.781903+0000","last_peered":"2024-09-22T03:50:26.781903+0000","last_clean":"2024-09-22T03:50:26.781903+0000","last_became_active":"2024-09-22T03:15:53.588564+0000","last_became_peered":"2024-09-22T03:15:53.588564+0000","last_unstale":"2024-09-22T03:50:26.781903+0000","last_undegraded":"2024-09-22T03:50:26.781903+0000","last_fullsized":"2024-09-22T03:50:26.781903+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":1334,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786601+0000","last_change":"2024-09-22T03:15:55.474905+0000","last_active":"2024-09-22T03:50:28.786601+0000","last_peered":"2024-09-22T03:50:28.786601+0000","last_clean":"2024-09-22T03:50:28.786601+0000","last_became_active":"2024-09-22T03:15:53.579172+0000","last_became_peered":"2024-09-22T03:15:53.579172+0000","last_unstale":"2024-09-22T03:50:28.786601+0000","last_undegraded":"2024-09-22T03:50:28.786601+0000","last_fullsized":"2024-09-22T03:50:28.786601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1332,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786303+0000","last_change":"2024-09-22T03:15:55.476632+0000","last_active":"2024-09-22T03:50:28.786303+0000","last_peered":"2024-09-22T03:50:28.786303+0000","last_clean":"2024-09-22T03:50:28.786303+0000","last_became_active":"2024-09-22T03:15:53.583253+0000","last_became_peered":"2024-09-22T03:15:53.583253+0000","last_unstale":"2024-09-22T03:50:28.786303+0000","last_undegraded":"2024-09-22T03:50:28.786303+0000","last_fullsized":"2024-09-22T03:50:28.786303+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1332,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786106+0000","last_change":"2024-09-22T03:15:55.473205+0000","last_active":"2024-09-22T03:50:28.786106+0000","last_peered":"2024-09-22T03:50:28.786106+0000","last_clean":"2024-09-22T03:50:28.786106+0000","last_became_active":"2024-09-22T03:15:53.577075+0000","last_became_peered":"2024-09-22T03:15:53.577075+0000","last_unstale":"2024-09-22T03:50:28.786106+0000","last_undegraded":"2024-09-22T03:50:28.786106+0000","last_fullsized":"2024-09-22T03:50:28.786106+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":1325,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.774629+0000","last_change":"2024-09-22T03:15:55.474940+0000","last_active":"2024-09-22T03:50:26.774629+0000","last_peered":"2024-09-22T03:50:26.774629+0000","last_clean":"2024-09-22T03:50:26.774629+0000","last_became_active":"2024-09-22T03:15:53.561718+0000","last_became_peered":"2024-09-22T03:15:53.561718+0000","last_unstale":"2024-09-22T03:50:26.774629+0000","last_undegraded":"2024-09-22T03:50:26.774629+0000","last_fullsized":"2024-09-22T03:50:26.774629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1453,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.780748+0000","last_change":"2024-09-22T03:15:52.494206+0000","last_active":"2024-09-22T03:50:26.780748+0000","last_peered":"2024-09-22T03:50:26.780748+0000","last_clean":"2024-09-22T03:50:26.780748+0000","last_became_active":"2024-09-22T03:15:52.482270+0000","last_became_peered":"2024-09-22T03:15:52.482270+0000","last_unstale":"2024-09-22T03:50:26.780748+0000","last_undegraded":"2024-09-22T03:50:26.780748+0000","last_fullsized":"2024-09-22T03:50:26.780748+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_clean_scrub_stamp":"2024-09-22T03:15:51.441048+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53527650304,"internally_reserved":0,"allocated":25223168,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160584077312,"internally_reserved":0,"allocated":74543104,"data_stored":74543104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359740441,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":24632,"kb_used_data":24632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52273096,"statfs":{"total":53552873472,"available":53527650304,"internally_reserved":0,"allocated":25223168,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359740445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":28124,"kb_used_data":28124,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52269604,"statfs":{"total":53552873472,"available":53524074496,"internally_reserved":0,"allocated":28798976,"data_stored":28798976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359740445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":19560,"kb_used_data":19560,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52278168,"statfs":{"total":53552873472,"available":53532844032,"internally_reserved":0,"allocated":20029440,"data_stored":20029440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53527650304,"internally_reserved":0,"allocated":25223168,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53532635136,"internally_reserved":0,"allocated":20238336,"data_stored":20238336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53527650304,"internally_reserved":0,"allocated":25223168,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53523791872,"internally_reserved":0,"allocated":29081600,"data_stored":29081600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:50:31.657 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:50:31.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:31.764+0000 7fef4c557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:31.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:31.771+0000 7fef4c557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:31.954 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:50:31.954 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-22T03:50:31.967 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1488,"stamp":"2024-09-22T03:50:30.768917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":72316,"kb_used_data":72316,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156820868,"statfs":{"total":160658620416,"available":160584568832,"internally_reserved":0,"allocated":74051584,"data_stored":74051584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4599727,"num_objects":-291,"num_object_clones":-3,"num_object_copies":-582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":-4931,"num_read_kb":-13685,"num_write":-929,"num_write_kb":-13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011033"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1324,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.779601+0000","last_change":"2024-09-22T03:15:55.475462+0000","last_active":"2024-09-22T03:50:26.779601+0000","last_peered":"2024-09-22T03:50:26.779601+0000","last_clean":"2024-09-22T03:50:26.779601+0000","last_became_active":"2024-09-22T03:15:53.559125+0000","last_became_peered":"2024-09-22T03:15:53.559125+0000","last_unstale":"2024-09-22T03:50:26.779601+0000","last_undegraded":"2024-09-22T03:50:26.779601+0000","last_fullsized":"2024-09-22T03:50:26.779601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1324,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.778916+0000","last_change":"2024-09-22T03:15:55.476029+0000","last_active":"2024-09-22T03:50:26.778916+0000","last_peered":"2024-09-22T03:50:26.778916+0000","last_clean":"2024-09-22T03:50:26.778916+0000","last_became_active":"2024-09-22T03:15:53.591886+0000","last_became_peered":"2024-09-22T03:15:53.591886+0000","last_unstale":"2024-09-22T03:50:26.778916+0000","last_undegraded":"2024-09-22T03:50:26.778916+0000","last_fullsized":"2024-09-22T03:50:26.778916+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1324,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.777102+0000","last_change":"2024-09-22T03:15:55.475380+0000","last_active":"2024-09-22T03:50:26.777102+0000","last_peered":"2024-09-22T03:50:26.777102+0000","last_clean":"2024-09-22T03:50:26.777102+0000","last_became_active":"2024-09-22T03:15:53.569486+0000","last_became_peered":"2024-09-22T03:15:53.569486+0000","last_unstale":"2024-09-22T03:50:26.777102+0000","last_undegraded":"2024-09-22T03:50:26.777102+0000","last_fullsized":"2024-09-22T03:50:26.777102+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1324,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.781903+0000","last_change":"2024-09-22T03:15:55.475811+0000","last_active":"2024-09-22T03:50:26.781903+0000","last_peered":"2024-09-22T03:50:26.781903+0000","last_clean":"2024-09-22T03:50:26.781903+0000","last_became_active":"2024-09-22T03:15:53.588564+0000","last_became_peered":"2024-09-22T03:15:53.588564+0000","last_unstale":"2024-09-22T03:50:26.781903+0000","last_undegraded":"2024-09-22T03:50:26.781903+0000","last_fullsized":"2024-09-22T03:50:26.781903+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":1334,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786601+0000","last_change":"2024-09-22T03:15:55.474905+0000","last_active":"2024-09-22T03:50:28.786601+0000","last_peered":"2024-09-22T03:50:28.786601+0000","last_clean":"2024-09-22T03:50:28.786601+0000","last_became_active":"2024-09-22T03:15:53.579172+0000","last_became_peered":"2024-09-22T03:15:53.579172+0000","last_unstale":"2024-09-22T03:50:28.786601+0000","last_undegraded":"2024-09-22T03:50:28.786601+0000","last_fullsized":"2024-09-22T03:50:28.786601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1332,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786303+0000","last_change":"2024-09-22T03:15:55.476632+0000","last_active":"2024-09-22T03:50:28.786303+0000","last_peered":"2024-09-22T03:50:28.786303+0000","last_clean":"2024-09-22T03:50:28.786303+0000","last_became_active":"2024-09-22T03:15:53.583253+0000","last_became_peered":"2024-09-22T03:15:53.583253+0000","last_unstale":"2024-09-22T03:50:28.786303+0000","last_undegraded":"2024-09-22T03:50:28.786303+0000","last_fullsized":"2024-09-22T03:50:28.786303+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1332,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786106+0000","last_change":"2024-09-22T03:15:55.473205+0000","last_active":"2024-09-22T03:50:28.786106+0000","last_peered":"2024-09-22T03:50:28.786106+0000","last_clean":"2024-09-22T03:50:28.786106+0000","last_became_active":"2024-09-22T03:15:53.577075+0000","last_became_peered":"2024-09-22T03:15:53.577075+0000","last_unstale":"2024-09-22T03:50:28.786106+0000","last_undegraded":"2024-09-22T03:50:28.786106+0000","last_fullsized":"2024-09-22T03:50:28.786106+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":1325,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.774629+0000","last_change":"2024-09-22T03:15:55.474940+0000","last_active":"2024-09-22T03:50:26.774629+0000","last_peered":"2024-09-22T03:50:26.774629+0000","last_clean":"2024-09-22T03:50:26.774629+0000","last_became_active":"2024-09-22T03:15:53.561718+0000","last_became_peered":"2024-09-22T03:15:53.561718+0000","last_unstale":"2024-09-22T03:50:26.774629+0000","last_undegraded":"2024-09-22T03:50:26.774629+0000","last_fullsized":"2024-09-22T03:50:26.774629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1453,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-09-22T03:50:26.780748+0000","last_change":"2024-09-22T03:15:52.494206+0000","last_active":"2024-09-22T03:50:26.780748+0000","last_peered":"2024-09-22T03:50:26.780748+0000","last_clean":"2024-09-22T03:50:26.780748+0000","last_became_active":"2024-09-22T03:15:52.482270+0000","last_became_peered":"2024-09-22T03:15:52.482270+0000","last_unstale":"2024-09-22T03:50:26.780748+0000","last_undegraded":"2024-09-22T03:50:26.780748+0000","last_fullsized":"2024-09-22T03:50:26.780748+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_clean_scrub_stamp":"2024-09-22T03:15:51.441048+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53527650304,"internally_reserved":0,"allocated":25223168,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160584077312,"internally_reserved":0,"allocated":74543104,"data_stored":74543104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359740441,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":24632,"kb_used_data":24632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52273096,"statfs":{"total":53552873472,"available":53527650304,"internally_reserved":0,"allocated":25223168,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359740445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":28124,"kb_used_data":28124,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52269604,"statfs":{"total":53552873472,"available":53524074496,"internally_reserved":0,"allocated":28798976,"data_stored":28798976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359740445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":19560,"kb_used_data":19560,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52278168,"statfs":{"total":53552873472,"available":53532844032,"internally_reserved":0,"allocated":20029440,"data_stored":20029440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53527650304,"internally_reserved":0,"allocated":25223168,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53532635136,"internally_reserved":0,"allocated":20238336,"data_stored":20238336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53527650304,"internally_reserved":0,"allocated":25223168,"data_stored":25223168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53523791872,"internally_reserved":0,"allocated":29081600,"data_stored":29081600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:50:31.968 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:50:31.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:50:32.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:32.067+0000 7fe229c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:32.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:32.073+0000 7fe229c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:32.247 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:50:32.248 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-22T03:50:32.260 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1489,"stamp":"2024-09-22T03:50:31.964647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":50628,"kb_used_data":50628,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156842556,"statfs":{"total":160658620416,"available":160606777344,"internally_reserved":0,"allocated":51843072,"data_stored":51843072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4664763,"num_objects":-326,"num_object_clones":-3,"num_object_copies":-652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":-5816,"num_read_kb":-14298,"num_write":-1190,"num_write_kb":-13465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.341644"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.809694+0000","last_change":"2024-09-22T03:15:55.475462+0000","last_active":"2024-09-22T03:50:30.809694+0000","last_peered":"2024-09-22T03:50:30.809694+0000","last_clean":"2024-09-22T03:50:30.809694+0000","last_became_active":"2024-09-22T03:15:53.559125+0000","last_became_peered":"2024-09-22T03:15:53.559125+0000","last_unstale":"2024-09-22T03:50:30.809694+0000","last_undegraded":"2024-09-22T03:50:30.809694+0000","last_fullsized":"2024-09-22T03:50:30.809694+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.815509+0000","last_change":"2024-09-22T03:15:55.476029+0000","last_active":"2024-09-22T03:50:30.815509+0000","last_peered":"2024-09-22T03:50:30.815509+0000","last_clean":"2024-09-22T03:50:30.815509+0000","last_became_active":"2024-09-22T03:15:53.591886+0000","last_became_peered":"2024-09-22T03:15:53.591886+0000","last_unstale":"2024-09-22T03:50:30.815509+0000","last_undegraded":"2024-09-22T03:50:30.815509+0000","last_fullsized":"2024-09-22T03:50:30.815509+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.813960+0000","last_change":"2024-09-22T03:15:55.475380+0000","last_active":"2024-09-22T03:50:30.813960+0000","last_peered":"2024-09-22T03:50:30.813960+0000","last_clean":"2024-09-22T03:50:30.813960+0000","last_became_active":"2024-09-22T03:15:53.569486+0000","last_became_peered":"2024-09-22T03:15:53.569486+0000","last_unstale":"2024-09-22T03:50:30.813960+0000","last_undegraded":"2024-09-22T03:50:30.813960+0000","last_fullsized":"2024-09-22T03:50:30.813960+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.820753+0000","last_change":"2024-09-22T03:15:55.475811+0000","last_active":"2024-09-22T03:50:30.820753+0000","last_peered":"2024-09-22T03:50:30.820753+0000","last_clean":"2024-09-22T03:50:30.820753+0000","last_became_active":"2024-09-22T03:15:53.588564+0000","last_became_peered":"2024-09-22T03:15:53.588564+0000","last_unstale":"2024-09-22T03:50:30.820753+0000","last_undegraded":"2024-09-22T03:50:30.820753+0000","last_fullsized":"2024-09-22T03:50:30.820753+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":1334,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786601+0000","last_change":"2024-09-22T03:15:55.474905+0000","last_active":"2024-09-22T03:50:28.786601+0000","last_peered":"2024-09-22T03:50:28.786601+0000","last_clean":"2024-09-22T03:50:28.786601+0000","last_became_active":"2024-09-22T03:15:53.579172+0000","last_became_peered":"2024-09-22T03:15:53.579172+0000","last_unstale":"2024-09-22T03:50:28.786601+0000","last_undegraded":"2024-09-22T03:50:28.786601+0000","last_fullsized":"2024-09-22T03:50:28.786601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1332,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786303+0000","last_change":"2024-09-22T03:15:55.476632+0000","last_active":"2024-09-22T03:50:28.786303+0000","last_peered":"2024-09-22T03:50:28.786303+0000","last_clean":"2024-09-22T03:50:28.786303+0000","last_became_active":"2024-09-22T03:15:53.583253+0000","last_became_peered":"2024-09-22T03:15:53.583253+0000","last_unstale":"2024-09-22T03:50:28.786303+0000","last_undegraded":"2024-09-22T03:50:28.786303+0000","last_fullsized":"2024-09-22T03:50:28.786303+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1332,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786106+0000","last_change":"2024-09-22T03:15:55.473205+0000","last_active":"2024-09-22T03:50:28.786106+0000","last_peered":"2024-09-22T03:50:28.786106+0000","last_clean":"2024-09-22T03:50:28.786106+0000","last_became_active":"2024-09-22T03:15:53.577075+0000","last_became_peered":"2024-09-22T03:15:53.577075+0000","last_unstale":"2024-09-22T03:50:28.786106+0000","last_undegraded":"2024-09-22T03:50:28.786106+0000","last_fullsized":"2024-09-22T03:50:28.786106+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":1337,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.820566+0000","last_change":"2024-09-22T03:15:55.474940+0000","last_active":"2024-09-22T03:50:30.820566+0000","last_peered":"2024-09-22T03:50:30.820566+0000","last_clean":"2024-09-22T03:50:30.820566+0000","last_became_active":"2024-09-22T03:15:53.561718+0000","last_became_peered":"2024-09-22T03:15:53.561718+0000","last_unstale":"2024-09-22T03:50:30.820566+0000","last_undegraded":"2024-09-22T03:50:30.820566+0000","last_fullsized":"2024-09-22T03:50:30.820566+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1465,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.814235+0000","last_change":"2024-09-22T03:15:52.494206+0000","last_active":"2024-09-22T03:50:30.814235+0000","last_peered":"2024-09-22T03:50:30.814235+0000","last_clean":"2024-09-22T03:50:30.814235+0000","last_became_active":"2024-09-22T03:15:52.482270+0000","last_became_peered":"2024-09-22T03:15:52.482270+0000","last_unstale":"2024-09-22T03:50:30.814235+0000","last_undegraded":"2024-09-22T03:50:30.814235+0000","last_fullsized":"2024-09-22T03:50:30.814235+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_clean_scrub_stamp":"2024-09-22T03:15:51.441048+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549858816,"internally_reserved":0,"allocated":3014656,"data_stored":3014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160606285824,"internally_reserved":0,"allocated":52334592,"data_stored":52334592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359740446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2944,"kb_used_data":2944,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294784,"statfs":{"total":53552873472,"available":53549858816,"internally_reserved":0,"allocated":3014656,"data_stored":3014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359740445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":28124,"kb_used_data":28124,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52269604,"statfs":{"total":53552873472,"available":53524074496,"internally_reserved":0,"allocated":28798976,"data_stored":28798976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359740445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":19560,"kb_used_data":19560,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52278168,"statfs":{"total":53552873472,"available":53532844032,"internally_reserved":0,"allocated":20029440,"data_stored":20029440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53549858816,"internally_reserved":0,"allocated":3014656,"data_stored":3014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53532635136,"internally_reserved":0,"allocated":20238336,"data_stored":20238336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549858816,"internally_reserved":0,"allocated":3014656,"data_stored":3014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53523791872,"internally_reserved":0,"allocated":29081600,"data_stored":29081600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:50:32.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:50:32.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:32.360+0000 7fdb7f06c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:32.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:32.366+0000 7fdb7f06c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:32.547 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:50:32.547 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":454,"fsid":"f1076ac7-a939-441d-8eeb-e35612888a01","created":"2024-09-22T03:15:42.327011+0000","modified":"2024-09-22T03:50:30.765330+0000","last_up_change":"2024-09-22T03:15:49.436070+0000","last_in_change":"2024-09-22T03:15:44.247845+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:15:51.416295+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:15:52.207511+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"adeb97d5-5208-4d0d-b820-010b67e342a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":421,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3847971085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6805","nonce":3847971085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6813","nonce":3847971085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3847971085}]},"public_addr":"172.21.15.70:6804/3847971085","cluster_addr":"172.21.15.70:6805/3847971085","heartbeat_back_addr":"172.21.15.70:6813/3847971085","heartbeat_front_addr":"172.21.15.70:6812/3847971085","state":["exists","up"]},{"osd":1,"uuid":"8f02f123-7480-4f9e-ab4a-b72de52061d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":421,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6803","nonce":3697286852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697286852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6807","nonce":3697286852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697286852}]},"public_addr":"172.21.15.70:6803/3697286852","cluster_addr":"172.21.15.70:6802/3697286852","heartbeat_back_addr":"172.21.15.70:6807/3697286852","heartbeat_front_addr":"172.21.15.70:6806/3697286852","state":["exists","up"]},{"osd":2,"uuid":"b4c0fde8-059a-4132-b1e7-872352bb8aa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":421,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":782143372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6809","nonce":782143372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6815","nonce":782143372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":782143372}]},"public_addr":"172.21.15.70:6810/782143372","cluster_addr":"172.21.15.70:6809/782143372","heartbeat_back_addr":"172.21.15.70:6815/782143372","heartbeat_front_addr":"172.21.15.70:6814/782143372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":7,"length":11},{"begin":23,"length":4},{"begin":37,"length":1},{"begin":41,"length":1},{"begin":44,"length":2},{"begin":69,"length":1},{"begin":71,"length":1}]},{"pool":4,"snaps":[{"begin":2,"length":1}]},{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":4}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":12,"snaps":[{"begin":2,"length":1}]},{"pool":13,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":2,"length":1}]},{"pool":16,"snaps":[{"begin":2,"length":1}]},{"pool":17,"snaps":[{"begin":2,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":2,"length":1}]},{"pool":20,"snaps":[{"begin":2,"length":1}]},{"pool":21,"snaps":[{"begin":2,"length":1}]},{"pool":22,"snaps":[{"begin":2,"length":3}]},{"pool":23,"snaps":[{"begin":2,"length":9}]},{"pool":24,"snaps":[{"begin":2,"length":1}]},{"pool":25,"snaps":[{"begin":2,"length":1}]},{"pool":26,"snaps":[{"begin":2,"length":1}]},{"pool":27,"snaps":[{"begin":2,"length":1}]},{"pool":28,"snaps":[{"begin":2,"length":1}]},{"pool":29,"snaps":[{"begin":2,"length":1}]},{"pool":30,"snaps":[{"begin":2,"length":1}]},{"pool":31,"snaps":[{"begin":2,"length":1}]},{"pool":32,"snaps":[{"begin":2,"length":1}]},{"pool":33,"snaps":[{"begin":2,"length":7},{"begin":14,"length":2},{"begin":19,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:50:33.561 INFO:tasks.ceph:Scrubbing osd.0 2024-09-22T03:50:33.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:50:33.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:33.673+0000 7efd68d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:33.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:33.684+0000 7efd68d17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:33.696 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-22T03:50:33.697 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-09-22T03:50:33.697 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-22T03:50:33.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-22T03:50:33.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:33.811+0000 7fbe12418640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:33.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:33.819+0000 7fbe12418640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.006 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 0 to deep-scrub 2024-09-22T03:50:34.024 INFO:tasks.ceph:Scrubbing osd.1 2024-09-22T03:50:34.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:50:34.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.124+0000 7f30d8d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.135+0000 7f30d8d97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.147 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-22T03:50:34.148 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-09-22T03:50:34.148 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-22T03:50:34.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-22T03:50:34.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.257+0000 7f00c8997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.264+0000 7f00c8997640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.438 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 1 to deep-scrub 2024-09-22T03:50:34.455 INFO:tasks.ceph:Scrubbing osd.2 2024-09-22T03:50:34.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-22T03:50:34.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.557+0000 7fbd78597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.567+0000 7fbd78597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.580 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-22T03:50:34.580 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "" 2024-09-22T03:50:34.580 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-22T03:50:34.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-22T03:50:34.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.690+0000 7f7a6f597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.698+0000 7f7a6f597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.876 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 2 to deep-scrub 2024-09-22T03:50:34.891 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:50:34.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.990+0000 7f04f6c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:34.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:34.997+0000 7f04f6c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:35.171 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:50:35.171 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-22T03:50:35.183 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1491,"stamp":"2024-09-22T03:50:33.964948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":50628,"kb_used_data":50628,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156842556,"statfs":{"total":160658620416,"available":160606777344,"internally_reserved":0,"allocated":51843072,"data_stored":51843072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4664763,"num_objects":-326,"num_object_clones":-3,"num_object_copies":-652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":-5816,"num_read_kb":-14298,"num_write":-1190,"num_write_kb":-13465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.204137"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.809694+0000","last_change":"2024-09-22T03:15:55.475462+0000","last_active":"2024-09-22T03:50:30.809694+0000","last_peered":"2024-09-22T03:50:30.809694+0000","last_clean":"2024-09-22T03:50:30.809694+0000","last_became_active":"2024-09-22T03:15:53.559125+0000","last_became_peered":"2024-09-22T03:15:53.559125+0000","last_unstale":"2024-09-22T03:50:30.809694+0000","last_undegraded":"2024-09-22T03:50:30.809694+0000","last_fullsized":"2024-09-22T03:50:30.809694+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.815509+0000","last_change":"2024-09-22T03:15:55.476029+0000","last_active":"2024-09-22T03:50:30.815509+0000","last_peered":"2024-09-22T03:50:30.815509+0000","last_clean":"2024-09-22T03:50:30.815509+0000","last_became_active":"2024-09-22T03:15:53.591886+0000","last_became_peered":"2024-09-22T03:15:53.591886+0000","last_unstale":"2024-09-22T03:50:30.815509+0000","last_undegraded":"2024-09-22T03:50:30.815509+0000","last_fullsized":"2024-09-22T03:50:30.815509+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.813960+0000","last_change":"2024-09-22T03:15:55.475380+0000","last_active":"2024-09-22T03:50:30.813960+0000","last_peered":"2024-09-22T03:50:30.813960+0000","last_clean":"2024-09-22T03:50:30.813960+0000","last_became_active":"2024-09-22T03:15:53.569486+0000","last_became_peered":"2024-09-22T03:15:53.569486+0000","last_unstale":"2024-09-22T03:50:30.813960+0000","last_undegraded":"2024-09-22T03:50:30.813960+0000","last_fullsized":"2024-09-22T03:50:30.813960+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1336,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.820753+0000","last_change":"2024-09-22T03:15:55.475811+0000","last_active":"2024-09-22T03:50:30.820753+0000","last_peered":"2024-09-22T03:50:30.820753+0000","last_clean":"2024-09-22T03:50:30.820753+0000","last_became_active":"2024-09-22T03:15:53.588564+0000","last_became_peered":"2024-09-22T03:15:53.588564+0000","last_unstale":"2024-09-22T03:50:30.820753+0000","last_undegraded":"2024-09-22T03:50:30.820753+0000","last_fullsized":"2024-09-22T03:50:30.820753+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":1334,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786601+0000","last_change":"2024-09-22T03:15:55.474905+0000","last_active":"2024-09-22T03:50:28.786601+0000","last_peered":"2024-09-22T03:50:28.786601+0000","last_clean":"2024-09-22T03:50:28.786601+0000","last_became_active":"2024-09-22T03:15:53.579172+0000","last_became_peered":"2024-09-22T03:15:53.579172+0000","last_unstale":"2024-09-22T03:50:28.786601+0000","last_undegraded":"2024-09-22T03:50:28.786601+0000","last_fullsized":"2024-09-22T03:50:28.786601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1332,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786303+0000","last_change":"2024-09-22T03:15:55.476632+0000","last_active":"2024-09-22T03:50:28.786303+0000","last_peered":"2024-09-22T03:50:28.786303+0000","last_clean":"2024-09-22T03:50:28.786303+0000","last_became_active":"2024-09-22T03:15:53.583253+0000","last_became_peered":"2024-09-22T03:15:53.583253+0000","last_unstale":"2024-09-22T03:50:28.786303+0000","last_undegraded":"2024-09-22T03:50:28.786303+0000","last_fullsized":"2024-09-22T03:50:28.786303+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1332,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-22T03:50:28.786106+0000","last_change":"2024-09-22T03:15:55.473205+0000","last_active":"2024-09-22T03:50:28.786106+0000","last_peered":"2024-09-22T03:50:28.786106+0000","last_clean":"2024-09-22T03:50:28.786106+0000","last_became_active":"2024-09-22T03:15:53.577075+0000","last_became_peered":"2024-09-22T03:15:53.577075+0000","last_unstale":"2024-09-22T03:50:28.786106+0000","last_undegraded":"2024-09-22T03:50:28.786106+0000","last_fullsized":"2024-09-22T03:50:28.786106+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":1337,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.820566+0000","last_change":"2024-09-22T03:15:55.474940+0000","last_active":"2024-09-22T03:50:30.820566+0000","last_peered":"2024-09-22T03:50:30.820566+0000","last_clean":"2024-09-22T03:50:30.820566+0000","last_became_active":"2024-09-22T03:15:53.561718+0000","last_became_peered":"2024-09-22T03:15:53.561718+0000","last_unstale":"2024-09-22T03:50:30.820566+0000","last_undegraded":"2024-09-22T03:50:30.820566+0000","last_fullsized":"2024-09-22T03:50:30.820566+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:52.444197+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1465,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-22T03:50:30.814235+0000","last_change":"2024-09-22T03:15:52.494206+0000","last_active":"2024-09-22T03:50:30.814235+0000","last_peered":"2024-09-22T03:50:30.814235+0000","last_clean":"2024-09-22T03:50:30.814235+0000","last_became_active":"2024-09-22T03:15:52.482270+0000","last_became_peered":"2024-09-22T03:15:52.482270+0000","last_unstale":"2024-09-22T03:50:30.814235+0000","last_undegraded":"2024-09-22T03:50:30.814235+0000","last_fullsized":"2024-09-22T03:50:30.814235+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:15:51.441048+0000","last_clean_scrub_stamp":"2024-09-22T03:15:51.441048+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549858816,"internally_reserved":0,"allocated":3014656,"data_stored":3014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160606285824,"internally_reserved":0,"allocated":52334592,"data_stored":52334592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359740446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2944,"kb_used_data":2944,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294784,"statfs":{"total":53552873472,"available":53549858816,"internally_reserved":0,"allocated":3014656,"data_stored":3014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359740445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":28124,"kb_used_data":28124,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52269604,"statfs":{"total":53552873472,"available":53524074496,"internally_reserved":0,"allocated":28798976,"data_stored":28798976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359740445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":19560,"kb_used_data":19560,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52278168,"statfs":{"total":53552873472,"available":53532844032,"internally_reserved":0,"allocated":20029440,"data_stored":20029440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53549858816,"internally_reserved":0,"allocated":3014656,"data_stored":3014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53532635136,"internally_reserved":0,"allocated":20238336,"data_stored":20238336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549858816,"internally_reserved":0,"allocated":3014656,"data_stored":3014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53523791872,"internally_reserved":0,"allocated":29081600,"data_stored":29081600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:50:35.186 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-22T03:15:52.444197+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=52, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.186 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-22T03:15:52.444197+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=52, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.186 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-22T03:15:52.444197+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=52, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.186 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-22T03:15:52.444197+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=52, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.186 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-22T03:15:52.444197+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=52, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.187 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-22T03:15:52.444197+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=52, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.187 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-22T03:15:52.444197+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=52, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.187 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-22T03:15:52.444197+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=52, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.187 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-22T03:15:51.441048+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=15, tm_sec=51, tm_wday=6, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=22, tm_hour=3, tm_min=50, tm_sec=32, tm_wday=6, tm_yday=266, tm_isdst=0) 2024-09-22T03:50:35.187 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-22T03:50:55.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:50:55.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:55.301+0000 7f9746b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:55.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:55.307+0000 7f9746b97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:55.480 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-22T03:50:55.480 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-09-22T03:50:55.493 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":1501,"stamp":"2024-09-22T03:50:53.968846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":7368,"kb_used_data":7368,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156885816,"statfs":{"total":160658620416,"available":160651075584,"internally_reserved":0,"allocated":7544832,"data_stored":7544832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002387"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1341,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.442068+0000","last_change":"2024-09-22T03:50:34.466256+0000","last_active":"2024-09-22T03:50:34.442068+0000","last_peered":"2024-09-22T03:50:34.442068+0000","last_clean":"2024-09-22T03:50:34.442068+0000","last_became_active":"2024-09-22T03:15:53.559125+0000","last_became_peered":"2024-09-22T03:15:53.559125+0000","last_unstale":"2024-09-22T03:50:34.442068+0000","last_undegraded":"2024-09-22T03:50:34.442068+0000","last_fullsized":"2024-09-22T03:50:34.442068+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:50:34.466168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.466168+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1341,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.442405+0000","last_change":"2024-09-22T03:50:34.463219+0000","last_active":"2024-09-22T03:50:34.442405+0000","last_peered":"2024-09-22T03:50:34.442405+0000","last_clean":"2024-09-22T03:50:34.442405+0000","last_became_active":"2024-09-22T03:15:53.591886+0000","last_became_peered":"2024-09-22T03:15:53.591886+0000","last_unstale":"2024-09-22T03:50:34.442405+0000","last_undegraded":"2024-09-22T03:50:34.442405+0000","last_fullsized":"2024-09-22T03:50:34.442405+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:50:34.463077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.463077+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1341,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.440944+0000","last_change":"2024-09-22T03:50:34.460438+0000","last_active":"2024-09-22T03:50:34.440944+0000","last_peered":"2024-09-22T03:50:34.440944+0000","last_clean":"2024-09-22T03:50:34.440944+0000","last_became_active":"2024-09-22T03:15:53.569486+0000","last_became_peered":"2024-09-22T03:15:53.569486+0000","last_unstale":"2024-09-22T03:50:34.440944+0000","last_undegraded":"2024-09-22T03:50:34.440944+0000","last_fullsized":"2024-09-22T03:50:34.440944+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:50:34.460343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.460343+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1341,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.440999+0000","last_change":"2024-09-22T03:50:34.458071+0000","last_active":"2024-09-22T03:50:34.440999+0000","last_peered":"2024-09-22T03:50:34.440999+0000","last_clean":"2024-09-22T03:50:34.440999+0000","last_became_active":"2024-09-22T03:15:53.588564+0000","last_became_peered":"2024-09-22T03:15:53.588564+0000","last_unstale":"2024-09-22T03:50:34.440999+0000","last_undegraded":"2024-09-22T03:50:34.440999+0000","last_fullsized":"2024-09-22T03:50:34.440999+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:50:34.457979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.457979+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"13'2","reported_seq":1345,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.007052+0000","last_change":"2024-09-22T03:50:34.021629+0000","last_active":"2024-09-22T03:50:34.007052+0000","last_peered":"2024-09-22T03:50:34.007052+0000","last_clean":"2024-09-22T03:50:34.007052+0000","last_became_active":"2024-09-22T03:15:53.579172+0000","last_became_peered":"2024-09-22T03:15:53.579172+0000","last_unstale":"2024-09-22T03:50:34.007052+0000","last_undegraded":"2024-09-22T03:50:34.007052+0000","last_fullsized":"2024-09-22T03:50:34.007052+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'2","last_scrub_stamp":"2024-09-22T03:50:34.021512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.021512+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1343,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.877509+0000","last_change":"2024-09-22T03:50:34.884805+0000","last_active":"2024-09-22T03:50:34.877509+0000","last_peered":"2024-09-22T03:50:34.877509+0000","last_clean":"2024-09-22T03:50:34.877509+0000","last_became_active":"2024-09-22T03:15:53.583253+0000","last_became_peered":"2024-09-22T03:15:53.583253+0000","last_unstale":"2024-09-22T03:50:34.877509+0000","last_undegraded":"2024-09-22T03:50:34.877509+0000","last_fullsized":"2024-09-22T03:50:34.877509+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:50:34.884714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.884714+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1343,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.876988+0000","last_change":"2024-09-22T03:50:34.884999+0000","last_active":"2024-09-22T03:50:34.876988+0000","last_peered":"2024-09-22T03:50:34.876988+0000","last_clean":"2024-09-22T03:50:34.876988+0000","last_became_active":"2024-09-22T03:15:53.577075+0000","last_became_peered":"2024-09-22T03:15:53.577075+0000","last_unstale":"2024-09-22T03:50:34.876988+0000","last_undegraded":"2024-09-22T03:50:34.876988+0000","last_fullsized":"2024-09-22T03:50:34.876988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:50:34.884921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.884921+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"11'1","reported_seq":1342,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.440065+0000","last_change":"2024-09-22T03:50:34.469608+0000","last_active":"2024-09-22T03:50:34.440065+0000","last_peered":"2024-09-22T03:50:34.440065+0000","last_clean":"2024-09-22T03:50:34.440065+0000","last_became_active":"2024-09-22T03:15:53.561718+0000","last_became_peered":"2024-09-22T03:15:53.561718+0000","last_unstale":"2024-09-22T03:50:34.440065+0000","last_undegraded":"2024-09-22T03:50:34.440065+0000","last_fullsized":"2024-09-22T03:50:34.440065+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"11'1","last_scrub_stamp":"2024-09-22T03:50:34.469506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.469506+0000","last_clean_scrub_stamp":"2024-09-22T03:15:52.444197+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1470,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-22T03:50:34.438996+0000","last_change":"2024-09-22T03:50:34.480709+0000","last_active":"2024-09-22T03:50:34.438996+0000","last_peered":"2024-09-22T03:50:34.438996+0000","last_clean":"2024-09-22T03:50:34.438996+0000","last_became_active":"2024-09-22T03:15:52.482270+0000","last_became_peered":"2024-09-22T03:15:52.482270+0000","last_unstale":"2024-09-22T03:50:34.438996+0000","last_undegraded":"2024-09-22T03:50:34.438996+0000","last_fullsized":"2024-09-22T03:50:34.438996+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"10'97","last_scrub_stamp":"2024-09-22T03:50:34.480567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:50:34.480567+0000","last_clean_scrub_stamp":"2024-09-22T03:15:51.441048+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549850624,"internally_reserved":0,"allocated":3022848,"data_stored":3022848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160651075584,"internally_reserved":0,"allocated":7544832,"data_stored":7544832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359740466,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2952,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294776,"statfs":{"total":53552873472,"available":53549850624,"internally_reserved":0,"allocated":3022848,"data_stored":3022848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359740465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1684,"kb_used_data":1684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296044,"statfs":{"total":53552873472,"available":53551149056,"internally_reserved":0,"allocated":1724416,"data_stored":1724416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359740465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2732,"kb_used_data":2732,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294996,"statfs":{"total":53552873472,"available":53550075904,"internally_reserved":0,"allocated":2797568,"data_stored":2797568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53549850624,"internally_reserved":0,"allocated":3022848,"data_stored":3022848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550075904,"internally_reserved":0,"allocated":2797568,"data_stored":2797568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549850624,"internally_reserved":0,"allocated":3022848,"data_stored":3022848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551149056,"internally_reserved":0,"allocated":1724416,"data_stored":1724416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:50:55.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-22T03:50:55.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:55.589+0000 7fec4e6ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:55.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-22T03:50:55.598+0000 7fec4e6ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:50:55.796 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T03:50:55.797 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T03:50:55.797 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:50:55.797 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:50:58.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T03:51:01.898 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:51:01.899 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:51:01.899 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:51:03.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:51:03.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T03:51:08.000 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:51:08.001 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:51:08.001 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:51:08.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:51:08.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:51:13.832 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-09-22T03:51:14.103 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:51:14.103 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T03:51:14.104 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-22T03:51:14.104 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:51:14.138 INFO:tasks.ceph.mgr.x:Stopped 2024-09-22T03:51:14.138 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T03:51:14.139 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-22T03:51:14.139 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:51:14.199 INFO:tasks.ceph.mon.a:Stopped 2024-09-22T03:51:14.199 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T03:51:14.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(POOL_FULL\)' | head -n 1 2024-09-22T03:51:14.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:51:14.283 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-22T03:51:14.479 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:51:14.479 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-22T03:51:14.551 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:51:14.551 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-22T03:51:14.609 INFO:tasks.ceph:Archiving mon data... 2024-09-22T03:51:14.610 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914750/data/mon.a.tgz 2024-09-22T03:51:14.610 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-09-22T03:51:14.626 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.9lAFhbpjGc 2024-09-22T03:51:14.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.9lAFhbpjGc 2024-09-22T03:51:14.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.9lAFhbpjGc 2024-09-22T03:51:14.837 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.9lAFhbpjGc is 281KB 2024-09-22T03:51:14.930 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.9lAFhbpjGc 2024-09-22T03:51:14.944 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T03:51:14.945 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:51:15.065 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:51:15.065 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T03:51:15.067 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914750/remote/smithi070/crash 2024-09-22T03:51:15.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:51:15.128 INFO:tasks.ceph:Compressing logs... 2024-09-22T03:51:15.129 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:51:15.193 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-22T03:51:15.193 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28596.log 2024-09-22T03:51:15.208 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73952.log 2024-09-22T03:51:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.28596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28347.log 2024-09-22T03:51:15.209 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28596.log.gz 2024-09-22T03:51:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.73952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29378.log 2024-09-22T03:51:15.209 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73952.log.gz 2024-09-22T03:51:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28347.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-22T03:51:15.209 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28347.log.gz 2024-09-22T03:51:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28765.log 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29378.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29378.log.gz 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38783.log 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29294.log 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28765.log.gz 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28958.log 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr: 80.2% -- replaced with /var/log/ceph/ceph-client.admin.38783.log.gz 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28387.log 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29294.log.gz 2024-09-22T03:51:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73709.log 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28958.log.gz 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29639.log 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.28387.log.gz 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.73709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28626.log 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr: 76.8% -- replaced with /var/log/ceph/ceph-client.admin.73709.log.gz 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28916.log 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28626.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28626.log.gz 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-22T03:51:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28438.log 2024-09-22T03:51:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28916.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28916.log.gz 2024-09-22T03:51:15.212 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.29639.log.gz 2024-09-22T03:51:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28846.log 2024-09-22T03:51:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73868.log 2024-09-22T03:51:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28438.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28438.log.gz 2024-09-22T03:51:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28846.log: 88.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-22T03:51:15.212 INFO:teuthology.orchestra.run.smithi070.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.28846.log.gz 2024-09-22T03:51:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29420.log 2024-09-22T03:51:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.73868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29208.log 2024-09-22T03:51:15.213 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.73868.log.gz 2024-09-22T03:51:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74036.log 2024-09-22T03:51:15.213 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29420.log.gz 2024-09-22T03:51:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29208.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29208.log.gzgzip 2024-09-22T03:51:15.213 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.74288.log 2024-09-22T03:51:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73775.log 2024-09-22T03:51:15.213 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.74036.log.gz 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29081.log 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74288.log.gz 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.73775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73910.log 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr: 81.3% -- replaced with /var/log/ceph/ceph-client.admin.73775.log.gz 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29081.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29081.log.gz 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74078.log 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.73910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44944.log 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73910.log.gz 2024-09-22T03:51:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28821.log 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74078.log.gz 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74246.log 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.44944.log.gz 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28821.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28821.log.gz 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56510.log 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74162.log 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74246.log.gz 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56510.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27201.log 2024-09-22T03:51:15.215 INFO:teuthology.orchestra.run.smithi070.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.56510.log.gz 2024-09-22T03:51:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74162.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-22T03:51:15.216 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.74162.log.gz 2024-09-22T03:51:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.tmp-client.admin.27201.log: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27201.log.gz 2024-09-22T03:51:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29206.log 2024-09-22T03:51:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-22T03:51:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29206.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29206.log.gz 2024-09-22T03:51:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73994.log 2024-09-22T03:51:15.216 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44872.log 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.73994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29080.log 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73994.log.gz 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29207.log 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.44872.log.gz 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29080.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29080.log.gz 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74330.log 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29207.log.gz 2024-09-22T03:51:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74120.log 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50808.log 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74330.log.gz 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29000.log 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.74120.log.gz 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50808.log: /var/log/ceph/ceph-client.admin.29000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74370.log 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29000.log.gz 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.50808.log.gz 2024-09-22T03:51:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67653.log 2024-09-22T03:51:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28237.log 2024-09-22T03:51:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74370.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74370.log.gz 2024-09-22T03:51:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.67653.log: 80.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28149.log 2024-09-22T03:51:15.219 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67653.log.gz 2024-09-22T03:51:15.219 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29336.log 2024-09-22T03:51:15.222 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28237.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28237.log.gz 2024-09-22T03:51:15.222 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.28149.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44803.log 2024-09-22T03:51:15.222 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28149.log.gz 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38714.log 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.29336.log: /var/log/ceph/ceph-client.admin.44803.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29336.log.gz 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr: 80.6% -- replaced with /var/log/ceph/ceph-client.admin.44803.log.gz 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29082.log 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.38714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74204.log 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr: 80.4%/var/log/ceph/ceph-client.admin.29082.log: -- replaced with /var/log/ceph/ceph-client.admin.38714.log.gz 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29082.log.gz 2024-09-22T03:51:15.223 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.74204.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.74204.log.gz 2024-09-22T03:51:15.371 INFO:teuthology.orchestra.run.smithi070.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-22T03:51:15.932 INFO:teuthology.orchestra.run.smithi070.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-22T03:51:18.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:51:22.892 INFO:teuthology.orchestra.run.smithi070.stderr: 95.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-22T03:51:22.987 INFO:teuthology.orchestra.run.smithi070.stderr: 95.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-22T03:51:25.295 INFO:teuthology.orchestra.run.smithi070.stderr: 96.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-22T03:51:25.297 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-22T03:51:25.297 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m10.113s 2024-09-22T03:51:25.297 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m25.351s 2024-09-22T03:51:25.297 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m1.128s 2024-09-22T03:51:25.298 INFO:tasks.ceph:Archiving logs... 2024-09-22T03:51:25.298 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914750/remote/smithi070/log 2024-09-22T03:51:25.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:51:28.258 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T03:51:28.266 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T03:51:28.267 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:51:28.311 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T03:51:28.325 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T03:51:28.325 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:51:28.348 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:51:28.352 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:51:28.352 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-09-22T03:51:28.352 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 -364us[ -358us] +/- 68ms 2024-09-22T03:51:28.353 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 +737us[ +743us] +/- 62ms 2024-09-22T03:51:28.353 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -490us[ -484us] +/- 77ms 2024-09-22T03:51:28.353 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:51:28.354 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T03:51:28.364 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T03:51:28.364 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T03:51:28.389 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:51:28.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:51:28.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:51:28.481 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T03:51:28.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T03:51:28.500 INFO:teuthology.task.internal:Duration was 2569.486713 seconds 2024-09-22T03:51:28.500 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T03:51:28.511 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T03:51:28.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:51:28.560 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:51:28.963 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T03:51:28.963 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-09-22T03:51:28.964 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:51:29.024 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T03:51:29.024 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:51:29.091 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T03:51:29.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:51:29.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T03:51:29.242 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T03:51:29.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:51:29.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T03:51:29.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:51:29.330 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-09-22T03:51:29.407 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:51:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:51:29.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T03:51:29.477 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T03:51:29.478 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914750/remote/smithi070 2024-09-22T03:51:29.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:51:29.534 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T03:51:29.535 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:51:29.584 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T03:51:29.594 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T03:51:29.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T03:51:29.604 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T03:51:29.604 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:51:29.638 INFO:teuthology.orchestra.run.smithi070.stdout: 265861 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 22 03:51 /home/ubuntu/cephtest 2024-09-22T03:51:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: 401541 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 22 03:15 /home/ubuntu/cephtest/mnt.0 2024-09-22T03:51:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: 401542 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 22 03:16 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-22T03:51:29.639 INFO:teuthology.orchestra.run.smithi070.stdout: 414062 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:16 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-22T03:51:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:51:29.640 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-22T03:51:29.640 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi070 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-22T03:51:29.646 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T03:51:29.657 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T03:51:29.669 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi070 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-22T03:51:29.673 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_api_tests} duration: 2569.4867131710052 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi070 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=58d1e69174f84835a4be824dc48d16ea status: fail success: false 2024-09-22T03:51:29.673 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:51:29.761 INFO:teuthology.run:FAIL