2024-04-16T07:33:29.392 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T07:33:29.393 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi --archive /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987 --owner scheduled_matan@teuthology --description crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} -- /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987/orig.config.yaml 2024-04-16T07:33:29.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T07:33:29.518 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987 branch: wip-matanb-crimson-replicated-april-15 description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} email: null first_in_suite: false job_id: '7657987' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-replicated-april-15 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(POOL_FULL\) sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: a7639cb6ce633dabc4d37769d62caacb54582984 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6444 sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: a7639cb6ce633dabc4d37769d62caacb54582984 targets: smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNb2mDReFOOnitsN7Mx+s7up9mh1GZXUj+FWZM1MBiYPxPvo1m4sNSkILYTUclrN6TJ6htpyremOhQ7ClHxDqfI= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_07:26:08 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T07:33:29.519 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa; will attempt to use it 2024-04-16T07:33:29.520 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks 2024-04-16T07:33:29.520 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T07:33:29.521 INFO:teuthology.task.internal:Checking packages... 2024-04-16T07:33:29.546 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '30b0aa0009a925d6a46536e55c5656edcd51ecdb' 2024-04-16T07:33:29.546 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T07:33:29.546 INFO:teuthology.packaging:ref: None 2024-04-16T07:33:29.546 INFO:teuthology.packaging:tag: None 2024-04-16T07:33:29.546 INFO:teuthology.packaging:branch: wip-matanb-crimson-replicated-april-15 2024-04-16T07:33:29.547 INFO:teuthology.packaging:sha1: 30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:33:29.547 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-replicated-april-15 2024-04-16T07:33:29.772 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3109.g30b0aa00 2024-04-16T07:33:29.773 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T07:33:29.780 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T07:33:29.780 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T07:33:29.803 INFO:teuthology.task.internal:Saving configuration 2024-04-16T07:33:29.817 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T07:33:29.828 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T07:33:29.856 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:27:17.555075', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNb2mDReFOOnitsN7Mx+s7up9mh1GZXUj+FWZM1MBiYPxPvo1m4sNSkILYTUclrN6TJ6htpyremOhQ7ClHxDqfI='} 2024-04-16T07:33:29.857 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T07:33:29.867 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-16T07:33:29.868 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T07:33:29.911 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff1036ade50>, signals=[15]) 2024-04-16T07:33:29.912 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T07:33:29.921 INFO:teuthology.task.internal:Opening connections... 2024-04-16T07:33:29.921 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T07:33:29.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:33:30.022 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T07:33:30.031 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-04-16T07:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-04-16T07:33:30.091 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-04-16T07:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2024-04-16T07:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T07:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-04-16T07:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T07:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T07:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T07:33:30.149 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-04-16T07:33:30.170 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T07:33:30.181 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T07:33:30.190 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T07:33:30.190 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-04-16T07:33:30.205 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T07:33:30.212 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T07:33:30.212 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-04-16T07:33:30.493 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T07:33:30.526 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T07:33:30.541 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T07:33:30.541 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T07:33:30.542 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T07:33:30.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:33:30.558 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-04-16T07:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-361.el9.x86_64 2024-04-16T07:33:30.615 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.14.0-361.el9.x86_64 2024-04-16T07:33:30.615 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2024-04-16T07:33:33.614 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-16T07:33:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-04-16T07:33:39.538 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-04-16T07:33:43.122 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-04-16T07:33:47.906 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.2 MB 00:01 2024-04-16T07:33:49.723 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-04-16T07:33:51.627 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T07:33:56.694 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-16T07:33:59.031 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T07:33:59.031 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T07:33:59.121 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:33:59.133 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:33:59.134 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:33:59.134 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:33:59.134 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:33:59.134 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T07:33:59.134 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:33:59.134 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T07:33:59.134 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T07:33:59.135 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T07:33:59.135 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:33:59.135 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:33:59.135 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:33:59.135 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2024-04-16T07:33:59.135 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:33:59.136 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 77 M 2024-04-16T07:33:59.136 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 123 M 2024-04-16T07:33:59.136 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:33:59.284 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.8 MB/s | 277 kB 00:00 2024-04-16T07:33:59.892 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-04-16T07:34:00.567 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 23 MB/s | 34 MB 00:01 2024-04-16T07:34:00.834 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 18 MB/s | 28 MB 00:01 2024-04-16T07:34:00.838 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:34:00.839 INFO:teuthology.orchestra.run.smithi118.stdout:Total 45 MB/s | 77 MB 00:01 2024-04-16T07:34:00.839 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:34:01.183 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:34:01.184 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:34:02.005 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:34:02.005 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:34:04.311 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:34:04.914 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:34:05.009 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:34:07.745 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:34:07.887 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:34:12.606 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:34:12.669 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:17.549 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:47.357 INFO:teuthology.orchestra.run.smithi118.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T07:34:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:34:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:47.763 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:47.763 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T07:34:47.764 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T07:34:47.764 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T07:34:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T07:34:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:34:48.203 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:34:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T07:34:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T07:34:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T07:34:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T07:34:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:34:48.204 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:34:48.582 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:34:48.676 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:34:48.676 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:34:48.677 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T07:34:48.677 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T07:34:48.677 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T07:34:48.677 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-16T07:34:48.677 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T07:34:48.677 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T07:34:48.758 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T07:34:48.759 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T07:34:48.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2024-04-16T07:34:48.792 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.06 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 68.el9 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Group : Unspecified 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 8264254 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T07:34:48.793 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : builder@centos.org 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:34:48.794 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T07:34:48.795 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T07:34:48.795 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 5.14.0-438.el9.x86_64 2024-04-16T07:34:48.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T07:34:49.412 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-04-16T07:34:50.856 INFO:teuthology.orchestra.run.smithi118.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T07:34:50.889 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-04-16T07:34:50.897 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-04-16T07:34:50.913 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.WLzCntDKq3 2024-04-16T07:34:50.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WLzCntDKq3 2024-04-16T07:34:50.983 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.WLzCntDKq3 2024-04-16T07:34:51.120 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.WLzCntDKq3 is 6KB 2024-04-16T07:34:51.169 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.WLzCntDKq3 2024-04-16T07:34:51.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T07:34:51.254 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T07:34:51.254 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T07:34:51.254 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T07:34:51.254 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T07:34:51.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T07:34:51.408 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-04-16T07:35:21.438 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-16T07:35:21.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-16T07:35:21.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:35:39.807 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-04-16T07:35:48.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-16T07:35:48.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:35:51.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-04-16T07:36:03.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-16T07:36:03.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:36:06.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-04-16T07:36:21.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-16T07:36:21.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:36:53.990 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-04-16T07:36:54.428 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-16T07:36:54.428 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T07:36:54.428 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-04-16T07:36:54.444 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-438.el9.x86_64 2024-04-16T07:36:54.444 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T07:36:54.444 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T07:36:54.444 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T07:36:55.445 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-16T07:36:55.446 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T07:36:55.537 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-04-16T07:36:55.567 DEBUG:teuthology.parallel:result is None 2024-04-16T07:36:55.567 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T07:36:55.576 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T07:36:55.576 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T07:36:55.605 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T07:36:55.616 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T07:36:55.630 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T07:36:55.630 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T07:36:55.711 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T07:36:55.722 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T07:36:55.723 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T07:36:55.815 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:36:55.842 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T07:36:55.845 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T07:36:55.857 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T07:36:55.857 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T07:36:55.905 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T07:36:55.918 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T07:36:55.918 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T07:36:55.982 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:36:56.087 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T07:36:56.165 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:36:56.219 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T07:36:56.284 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:36:56.284 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T07:36:56.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-04-16T07:36:56.444 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T07:36:56.815 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T07:36:56.828 INFO:teuthology.task.internal:Starting timer... 2024-04-16T07:36:56.828 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T07:36:56.845 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T07:36:56.860 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T07:36:56.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-04-16T07:36:57.007 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-04-16T07:36:57.009 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T07:36:57.009 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-04-16T07:36:57.029 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-04-16T07:36:57.029 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2024-04-16T07:36:57.030 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T07:36:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:36:57.103 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T07:36:57.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-04-16T07:36:57.242 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T07:36:57.252 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T07:36:57.275 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T07:36:57.276 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T07:41:29.327 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2024-04-16T07:41:29.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-16T07:41:29.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T07:41:29.428 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-04-16T07:41:29.501 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-04-16T07:41:29.502 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T07:41:29.512 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T07:41:29.513 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T07:41:29.513 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T07:41:29.574 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T07:41:29.591 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T07:41:29.622 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-04-16T07:41:29.635 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-04-16T07:41:29.652 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T07:41:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T07:41:29.725 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-04-16T07:41:29.729 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T07:41:29.729 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-04-16T07:41:29.729 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:29.729 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:29.729 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:29.729 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T07:41:29.729 INFO:teuthology.run_tasks:Running task install... 2024-04-16T07:41:29.738 DEBUG:teuthology.task.install:project ceph 2024-04-16T07:41:29.738 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}} 2024-04-16T07:41:29.738 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'} 2024-04-16T07:41:29.738 INFO:teuthology.task.install:Using flavor: crimson 2024-04-16T07:41:29.744 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T07:41:29.744 INFO:teuthology.task.install:extra packages: [] 2024-04-16T07:41:29.744 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'tag': None, 'wait_for_package': False} 2024-04-16T07:41:29.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:41:29.887 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/ 2024-04-16T07:41:29.887 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3109.g30b0aa00 2024-04-16T07:41:30.613 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-replicated-april-15/30b0aa0009a925d6a46536e55c5656edcd51ecdb/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T07:41:30.613 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:41:30.614 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T07:41:30.647 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T07:41:30.647 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/30b0aa0009a925d6a46536e55c5656edcd51ecdb/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T07:41:30.719 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T07:41:30.804 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T07:41:30.832 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-04-16T07:41:30.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-04-16T07:41:31.083 INFO:teuthology.orchestra.run.smithi118.stdout:74 files removed 2024-04-16T07:41:31.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2024-04-16T07:41:32.519 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 79 kB/s | 87 kB 00:01 2024-04-16T07:41:34.403 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 6.3 kB/s | 12 kB 00:01 2024-04-16T07:41:35.254 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 2.5 kB/s | 2.0 kB 00:00 2024-04-16T07:41:35.681 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-04-16T07:41:37.165 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-04-16T07:41:38.614 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.0 MB 00:01 2024-04-16T07:41:43.926 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-04-16T07:41:55.865 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 750 kB/s | 6.2 MB 00:08 2024-04-16T07:41:57.532 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-16T07:41:59.259 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-16T07:42:04.209 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-16T07:42:06.962 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 10 M 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.7 M 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T07:42:06.967 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.9 M 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 19 M 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 25 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 797 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 546 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.2 M 2024-04-16T07:42:06.968 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 45 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 135 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 159 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 315 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 293 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 99 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T07:42:06.969 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout:Install 30 Packages 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:42:06.970 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 59 M 2024-04-16T07:42:06.971 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:42:15.318 INFO:teuthology.orchestra.run.smithi118.stdout:(1/32): ceph-base-19.0.0-3109.g30b0aa00.el9.x86 760 kB/s | 5.9 MB 00:07 2024-04-16T07:42:15.693 INFO:teuthology.orchestra.run.smithi118.stdout:(2/32): ceph-selinux-19.0.0-3109.g30b0aa00.el9. 67 kB/s | 25 kB 00:00 2024-04-16T07:42:16.893 INFO:teuthology.orchestra.run.smithi118.stdout:(3/32): libcephfs2-19.0.0-3109.g30b0aa00.el9.x8 664 kB/s | 797 kB 00:01 2024-04-16T07:42:17.801 INFO:teuthology.orchestra.run.smithi118.stdout:(4/32): libradosstriper1-19.0.0-3109.g30b0aa00. 601 kB/s | 546 kB 00:00 2024-04-16T07:42:32.501 INFO:teuthology.orchestra.run.smithi118.stdout:(5/32): librgw2-19.0.0-3109.g30b0aa00.el9.x86_6 365 kB/s | 5.2 MB 00:14 2024-04-16T07:42:32.675 INFO:teuthology.orchestra.run.smithi118.stdout:(6/32): python3-ceph-argparse-19.0.0-3109.g30b0 258 kB/s | 45 kB 00:00 2024-04-16T07:42:33.067 INFO:teuthology.orchestra.run.smithi118.stdout:(7/32): python3-ceph-common-19.0.0-3109.g30b0aa 344 kB/s | 135 kB 00:00 2024-04-16T07:42:33.634 INFO:teuthology.orchestra.run.smithi118.stdout:(8/32): ceph-radosgw-19.0.0-3109.g30b0aa00.el9. 394 kB/s | 10 MB 00:26 2024-04-16T07:42:33.776 INFO:teuthology.orchestra.run.smithi118.stdout:(9/32): python3-cephfs-19.0.0-3109.g30b0aa00.el 225 kB/s | 159 kB 00:00 2024-04-16T07:42:34.568 INFO:teuthology.orchestra.run.smithi118.stdout:(10/32): python3-rbd-19.0.0-3109.g30b0aa00.el9. 370 kB/s | 293 kB 00:00 2024-04-16T07:42:34.777 INFO:teuthology.orchestra.run.smithi118.stdout:(11/32): python3-rados-19.0.0-3109.g30b0aa00.el 276 kB/s | 315 kB 00:01 2024-04-16T07:42:34.840 INFO:teuthology.orchestra.run.smithi118.stdout:(12/32): python3-rgw-19.0.0-3109.g30b0aa00.el9. 366 kB/s | 99 kB 00:00 2024-04-16T07:42:34.944 INFO:teuthology.orchestra.run.smithi118.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 635 kB/s | 106 kB 00:00 2024-04-16T07:42:34.978 INFO:teuthology.orchestra.run.smithi118.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-04-16T07:42:35.003 INFO:teuthology.orchestra.run.smithi118.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 757 kB/s | 44 kB 00:00 2024-04-16T07:42:35.095 INFO:teuthology.orchestra.run.smithi118.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 661 kB 00:00 2024-04-16T07:42:35.121 INFO:teuthology.orchestra.run.smithi118.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-04-16T07:42:35.154 INFO:teuthology.orchestra.run.smithi118.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 710 kB/s | 41 kB 00:00 2024-04-16T07:42:35.180 INFO:teuthology.orchestra.run.smithi118.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-04-16T07:42:35.205 INFO:teuthology.orchestra.run.smithi118.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 698 kB/s | 35 kB 00:00 2024-04-16T07:42:35.230 INFO:teuthology.orchestra.run.smithi118.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 924 kB/s | 46 kB 00:00 2024-04-16T07:42:35.497 INFO:teuthology.orchestra.run.smithi118.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-04-16T07:42:35.564 INFO:teuthology.orchestra.run.smithi118.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-04-16T07:42:35.631 INFO:teuthology.orchestra.run.smithi118.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-04-16T07:42:35.715 INFO:teuthology.orchestra.run.smithi118.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-04-16T07:42:35.740 INFO:teuthology.orchestra.run.smithi118.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-04-16T07:42:35.790 INFO:teuthology.orchestra.run.smithi118.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-04-16T07:42:35.818 INFO:teuthology.orchestra.run.smithi118.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.4 MB/s | 192 kB 00:00 2024-04-16T07:42:35.883 INFO:teuthology.orchestra.run.smithi118.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-16T07:42:41.816 INFO:teuthology.orchestra.run.smithi118.stdout:(30/32): librados2-19.0.0-3109.g30b0aa00.el9.x8 580 kB/s | 3.4 MB 00:05 2024-04-16T07:42:45.499 INFO:teuthology.orchestra.run.smithi118.stdout:(31/32): librbd1-19.0.0-3109.g30b0aa00.el9.x86_ 391 kB/s | 3.7 MB 00:09 2024-04-16T07:43:09.930 INFO:teuthology.orchestra.run.smithi118.stdout:(32/32): ceph-common-19.0.0-3109.g30b0aa00.el9. 318 kB/s | 19 MB 01:02 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:43:09.932 INFO:teuthology.orchestra.run.smithi118.stdout:Total 962 kB/s | 59 MB 01:02 2024-04-16T07:43:09.933 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:43:09.968 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:43:09.968 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:43:10.460 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:43:10.460 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:43:11.943 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:43:12.448 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T07:43:12.507 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:43:12.597 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:43:12.686 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:43:12.718 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:43:12.779 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:43:12.824 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T07:43:12.863 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T07:43:12.896 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T07:43:12.922 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:43:12.935 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T07:43:13.113 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:43:13.152 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T07:43:13.504 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T07:43:13.532 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:43:13.624 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:43:13.666 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 12/34 2024-04-16T07:43:13.720 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:43:13.746 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 14/34 2024-04-16T07:43:13.824 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T07:43:13.851 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:43:13.939 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 16/34 2024-04-16T07:43:13.975 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T07:43:14.081 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T07:43:14.111 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T07:43:14.151 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T07:43:14.183 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T07:43:14.212 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T07:43:14.243 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T07:43:14.287 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T07:43:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T07:43:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:43:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 26/34 2024-04-16T07:43:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 27/34 2024-04-16T07:43:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 28/34 2024-04-16T07:43:17.394 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:43:17.441 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:43:18.144 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 29/34 2024-04-16T07:43:18.195 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:43:18.269 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 30/34 2024-04-16T07:43:18.269 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T07:43:18.269 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T07:43:18.270 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:43:18.305 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:43:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:43:28.526 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-04-16T07:43:28.526 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-04-16T07:43:28.526 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-04-16T07:43:28.526 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-04-16T07:43:28.526 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-04-16T07:43:28.526 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2024-04-16T07:43:28.526 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-04-16T07:43:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:43:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:43:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/34 2024-04-16T07:43:29.647 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:43:29.647 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T07:43:29.647 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:43:29.648 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T07:43:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:43:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:43:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T07:43:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:43:31.033 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:43:31.033 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/34 2024-04-16T07:43:31.033 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/34 2024-04-16T07:43:31.033 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_6 6/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 7/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9. 8/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x8 9/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 12/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 13/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T07:43:31.034 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T07:43:31.035 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T07:43:31.036 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T07:43:31.036 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 31/34 2024-04-16T07:43:31.036 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T07:43:31.036 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/34 2024-04-16T07:43:32.171 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T07:43:32.172 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.173 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:43:32.174 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:43:32.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2024-04-16T07:43:32.993 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:28 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:43:33.144 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:43:33.145 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:43:33.145 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:43:33.145 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:43:33.145 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:43:33.145 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 84 M 2024-04-16T07:43:33.145 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:43:33.145 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 173 k 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T07:43:33.146 INFO:teuthology.orchestra.run.smithi118.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T07:43:33.147 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:43:33.147 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:43:33.147 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:43:33.147 INFO:teuthology.orchestra.run.smithi118.stdout:Install 11 Packages 2024-04-16T07:43:33.147 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:43:33.147 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 87 M 2024-04-16T07:43:33.147 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 814 M 2024-04-16T07:43:33.148 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:43:33.763 INFO:teuthology.orchestra.run.smithi118.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-04-16T07:43:33.955 INFO:teuthology.orchestra.run.smithi118.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-04-16T07:43:33.997 INFO:teuthology.orchestra.run.smithi118.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-04-16T07:43:34.030 INFO:teuthology.orchestra.run.smithi118.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-04-16T07:43:34.072 INFO:teuthology.orchestra.run.smithi118.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.1 MB/s | 90 kB 00:00 2024-04-16T07:43:34.731 INFO:teuthology.orchestra.run.smithi118.stdout:(6/11): libcephsqlite-19.0.0-3109.g30b0aa00.el9 153 kB/s | 173 kB 00:01 2024-04-16T07:43:34.982 INFO:teuthology.orchestra.run.smithi118.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 206 kB/s | 187 kB 00:00 2024-04-16T07:43:35.415 INFO:teuthology.orchestra.run.smithi118.stdout:(8/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 686 kB/s | 469 kB 00:00 2024-04-16T07:43:35.482 INFO:teuthology.orchestra.run.smithi118.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 373 kB/s | 187 kB 00:00 2024-04-16T07:43:35.807 INFO:teuthology.orchestra.run.smithi118.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 311 kB/s | 122 kB 00:00 2024-04-16T07:46:14.004 INFO:teuthology.orchestra.run.smithi118.stdout:(11/11): ceph-test-19.0.0-3109.g30b0aa00.el9.x8 538 kB/s | 84 MB 02:40 2024-04-16T07:46:14.006 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:46:14.006 INFO:teuthology.orchestra.run.smithi118.stdout:Total 554 kB/s | 87 MB 02:40 2024-04-16T07:46:14.040 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:46:14.062 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:46:14.062 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:46:14.684 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:46:14.684 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:46:15.710 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:46:15.770 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T07:46:15.812 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T07:46:15.846 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:46:15.927 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T07:46:15.971 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T07:46:16.015 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T07:46:16.060 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T07:46:16.091 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T07:46:16.145 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T07:46:16.222 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T07:46:16.252 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:46:27.785 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 10/11 2024-04-16T07:46:27.860 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:46:34.349 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 11/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T07:46:34.350 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T07:46:34.778 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T07:46:34.778 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:46:34.778 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:46:34.778 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:46:34.778 INFO:teuthology.orchestra.run.smithi118.stdout: jq-1.6-16.el9.x86_64 2024-04-16T07:46:34.778 INFO:teuthology.orchestra.run.smithi118.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:46:34.779 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:46:35.078 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2024-04-16T07:46:35.665 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:31 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:46:35.821 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 6.5 k 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 2.2 M 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 1.6 M 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 249 k 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 5.0 M 2024-04-16T07:46:35.824 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 71 M 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T07:46:35.825 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T07:46:35.826 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout:Install 36 Packages 2024-04-16T07:46:35.827 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:46:35.828 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 84 M 2024-04-16T07:46:35.828 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 922 M 2024-04-16T07:46:35.828 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:46:36.945 INFO:teuthology.orchestra.run.smithi118.stdout:(1/36): ceph-19.0.0-3109.g30b0aa00.el9.x86_64.r 13 kB/s | 6.5 kB 00:00 2024-04-16T07:46:39.103 INFO:teuthology.orchestra.run.smithi118.stdout:(2/36): ceph-mgr-19.0.0-3109.g30b0aa00.el9.x86_ 601 kB/s | 1.6 MB 00:02 2024-04-16T07:46:42.937 INFO:teuthology.orchestra.run.smithi118.stdout:(3/36): ceph-mds-19.0.0-3109.g30b0aa00.el9.x86_ 345 kB/s | 2.2 MB 00:06 2024-04-16T07:46:43.420 INFO:teuthology.orchestra.run.smithi118.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3109.g30b0 515 kB/s | 249 kB 00:00 2024-04-16T07:46:43.545 INFO:teuthology.orchestra.run.smithi118.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-04-16T07:46:43.587 INFO:teuthology.orchestra.run.smithi118.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T07:46:43.621 INFO:teuthology.orchestra.run.smithi118.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-04-16T07:46:43.654 INFO:teuthology.orchestra.run.smithi118.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T07:46:43.688 INFO:teuthology.orchestra.run.smithi118.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T07:46:43.730 INFO:teuthology.orchestra.run.smithi118.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-04-16T07:46:43.805 INFO:teuthology.orchestra.run.smithi118.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-04-16T07:46:43.839 INFO:teuthology.orchestra.run.smithi118.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-04-16T07:46:43.881 INFO:teuthology.orchestra.run.smithi118.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-04-16T07:46:43.923 INFO:teuthology.orchestra.run.smithi118.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-04-16T07:46:44.065 INFO:teuthology.orchestra.run.smithi118.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 882 kB/s | 125 kB 00:00 2024-04-16T07:46:44.124 INFO:teuthology.orchestra.run.smithi118.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 383 kB/s | 22 kB 00:00 2024-04-16T07:46:44.299 INFO:teuthology.orchestra.run.smithi118.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 863 kB/s | 151 kB 00:00 2024-04-16T07:46:44.341 INFO:teuthology.orchestra.run.smithi118.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-04-16T07:46:44.408 INFO:teuthology.orchestra.run.smithi118.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-16T07:46:44.500 INFO:teuthology.orchestra.run.smithi118.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-04-16T07:46:44.542 INFO:teuthology.orchestra.run.smithi118.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-04-16T07:46:44.592 INFO:teuthology.orchestra.run.smithi118.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-04-16T07:46:44.643 INFO:teuthology.orchestra.run.smithi118.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-04-16T07:46:44.693 INFO:teuthology.orchestra.run.smithi118.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-04-16T07:46:44.743 INFO:teuthology.orchestra.run.smithi118.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-04-16T07:46:44.802 INFO:teuthology.orchestra.run.smithi118.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 794 kB/s | 46 kB 00:00 2024-04-16T07:46:44.853 INFO:teuthology.orchestra.run.smithi118.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-16T07:46:44.911 INFO:teuthology.orchestra.run.smithi118.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-04-16T07:46:44.961 INFO:teuthology.orchestra.run.smithi118.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-04-16T07:46:45.012 INFO:teuthology.orchestra.run.smithi118.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-04-16T07:46:45.062 INFO:teuthology.orchestra.run.smithi118.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 717 kB/s | 36 kB 00:00 2024-04-16T07:46:45.121 INFO:teuthology.orchestra.run.smithi118.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-04-16T07:46:45.188 INFO:teuthology.orchestra.run.smithi118.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2024-04-16T07:46:45.238 INFO:teuthology.orchestra.run.smithi118.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-04-16T07:46:48.853 INFO:teuthology.orchestra.run.smithi118.stdout:(35/36): ceph-mon-19.0.0-3109.g30b0aa00.el9.x86 432 kB/s | 5.0 MB 00:11 2024-04-16T07:47:37.483 INFO:teuthology.orchestra.run.smithi118.stdout:(36/36): ceph-osd-19.0.0-3109.g30b0aa00.el9.x86 1.2 MB/s | 71 MB 00:58 2024-04-16T07:47:37.485 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:47:37.485 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.4 MB/s | 84 MB 01:01 2024-04-16T07:47:37.504 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:47:37.542 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:47:37.543 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:47:38.274 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:47:38.274 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:47:39.864 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:47:39.934 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T07:47:39.987 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T07:47:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T07:47:40.090 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T07:47:40.144 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T07:47:40.177 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T07:47:40.268 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T07:47:40.355 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T07:47:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T07:47:40.468 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T07:47:40.534 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T07:47:40.569 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T07:47:40.627 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T07:47:40.689 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T07:47:40.884 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T07:47:40.932 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T07:47:40.968 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T07:47:41.006 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T07:47:41.044 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T07:47:41.128 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T07:47:41.171 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:47:41.205 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T07:47:41.245 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T07:47:41.286 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T07:47:41.329 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T07:47:41.426 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T07:47:41.460 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T07:47:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T07:47:41.693 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T07:47:41.820 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:47:42.014 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 31/36 2024-04-16T07:47:42.041 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:47:42.066 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 32/36 2024-04-16T07:47:42.066 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:47:42.066 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T07:47:42.066 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:47:42.067 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T07:47:42.067 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:47:52.883 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:47:52.911 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 33/36 2024-04-16T07:47:52.911 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:47:52.912 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T07:47:52.912 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:47:52.912 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T07:47:52.912 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:47:53.443 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:47:53.465 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 34/36 2024-04-16T07:47:53.466 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:47:53.466 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T07:47:53.466 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:47:53.466 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T07:47:53.466 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:47:53.735 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:47:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 35/36 2024-04-16T07:47:53.758 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:47:53.758 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T07:47:53.758 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:47:53.759 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T07:47:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:47:53.824 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 36/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 3/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 4/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 5/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9. 6/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T07:48:01.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T07:48:01.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T07:48:01.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T07:48:02.572 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T07:48:02.572 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:02.572 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:02.572 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:02.572 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:02.572 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:02.572 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T07:48:02.573 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T07:48:02.574 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T07:48:02.575 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T07:48:02.575 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T07:48:02.575 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T07:48:02.575 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T07:48:02.575 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:02.575 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:02.838 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2024-04-16T07:48:03.407 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:05:59 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:03.504 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:03.555 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:03.556 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:03.557 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:03.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2024-04-16T07:48:04.190 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:00 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:04.344 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:04.345 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:04.345 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:48:04.345 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:04.345 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:04.345 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 762 k 2024-04-16T07:48:04.345 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:04.345 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:04.346 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:04.346 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-04-16T07:48:04.346 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:04.346 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 762 k 2024-04-16T07:48:04.346 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 769 k 2024-04-16T07:48:04.346 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:04.777 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-19.0.0-3109.g30b0aa00.el9.noarch.rpm 1.7 MB/s | 762 kB 00:00 2024-04-16T07:48:04.777 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:04.778 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-04-16T07:48:04.778 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:04.783 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:04.783 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:04.789 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:04.789 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:04.866 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:48:05.464 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:48:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:48:06.147 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/1 2024-04-16T07:48:06.147 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:06.147 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:06.148 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:06.148 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:06.148 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:06.401 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T07:48:06.990 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:02 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:07.144 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:07.145 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-04-16T07:48:07.145 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:07.145 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:07.145 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 157 k 2024-04-16T07:48:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:07.146 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:07.146 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:07.146 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-04-16T07:48:07.146 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:07.146 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 157 k 2024-04-16T07:48:07.146 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 640 k 2024-04-16T07:48:07.146 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:07.451 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-19.0.0-3109.g30b0aa 515 kB/s | 157 kB 00:00 2024-04-16T07:48:07.452 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:07.452 INFO:teuthology.orchestra.run.smithi118.stdout:Total 513 kB/s | 157 kB 00:00 2024-04-16T07:48:07.452 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:07.462 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:07.462 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:07.515 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:07.515 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:07.666 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:07.719 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:48:07.744 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:48:07.744 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:48:07.744 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T07:48:07.744 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:08.738 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00. 1/1 2024-04-16T07:48:08.739 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:08.739 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:08.739 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:08.739 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:08.739 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:08.979 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2024-04-16T07:48:09.557 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:05 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:09.655 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:09.706 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:09.707 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:09.708 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:09.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T07:48:10.358 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:06 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:10.511 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 3.5 M 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 34 k 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 16 k 2024-04-16T07:48:10.512 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout:Install 11 Packages 2024-04-16T07:48:10.513 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:10.514 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 7.6 M 2024-04-16T07:48:10.514 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 100 M 2024-04-16T07:48:10.514 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:11.365 INFO:teuthology.orchestra.run.smithi118.stdout:(1/11): ceph-grafana-dashboards-19.0.0-3109.g30 186 kB/s | 34 kB 00:00 2024-04-16T07:48:11.490 INFO:teuthology.orchestra.run.smithi118.stdout:(2/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-04-16T07:48:11.640 INFO:teuthology.orchestra.run.smithi118.stdout:(3/11): ceph-mgr-dashboard-19.0.0-3109.g30b0aa0 7.7 MB/s | 3.5 MB 00:00 2024-04-16T07:48:11.699 INFO:teuthology.orchestra.run.smithi118.stdout:(4/11): ceph-prometheus-alerts-19.0.0-3109.g30b 32 kB/s | 16 kB 00:00 2024-04-16T07:48:11.816 INFO:teuthology.orchestra.run.smithi118.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-04-16T07:48:11.874 INFO:teuthology.orchestra.run.smithi118.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.3 MB/s | 548 kB 00:00 2024-04-16T07:48:11.925 INFO:teuthology.orchestra.run.smithi118.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-04-16T07:48:11.975 INFO:teuthology.orchestra.run.smithi118.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 617 kB/s | 31 kB 00:00 2024-04-16T07:48:12.033 INFO:teuthology.orchestra.run.smithi118.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-04-16T07:48:12.076 INFO:teuthology.orchestra.run.smithi118.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.9 MB/s | 2.0 MB 00:00 2024-04-16T07:48:13.184 INFO:teuthology.orchestra.run.smithi118.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 510 kB/s | 863 kB 00:01 2024-04-16T07:48:13.188 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:13.188 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.8 MB/s | 7.6 MB 00:02 2024-04-16T07:48:13.209 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:13.228 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:13.228 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:13.407 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:13.407 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:13.882 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:13.965 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T07:48:13.995 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T07:48:14.085 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T07:48:14.212 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T07:48:14.266 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T07:48:14.350 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T07:48:14.401 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T07:48:14.430 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:48:14.462 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 9/11 2024-04-16T07:48:15.509 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 10/11 2024-04-16T07:48:15.546 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:48:17.101 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 11/11 2024-04-16T07:48:17.101 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el 1/11 2024-04-16T07:48:17.101 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noa 2/11 2024-04-16T07:48:17.101 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9 3/11 2024-04-16T07:48:17.101 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T07:48:17.101 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T07:48:17.101 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T07:48:17.101 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T07:48:17.102 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T07:48:17.102 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T07:48:17.102 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T07:48:17.470 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T07:48:17.471 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T07:48:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T07:48:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T07:48:17.472 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:17.472 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:17.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T07:48:18.339 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:14 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:18.491 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 7.4 M 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T07:48:18.493 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T07:48:18.494 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T07:48:18.494 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T07:48:18.494 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T07:48:18.494 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:18.494 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:18.494 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-04-16T07:48:18.494 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2024-04-16T07:48:18.494 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:18.495 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 37 M 2024-04-16T07:48:18.495 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 209 M 2024-04-16T07:48:18.495 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:18.851 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 176 kB/s | 32 kB 00:00 2024-04-16T07:48:18.885 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 522 kB/s | 17 kB 00:00 2024-04-16T07:48:18.926 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 822 kB/s | 34 kB 00:00 2024-04-16T07:48:19.027 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.4 MB/s | 3.0 MB 00:00 2024-04-16T07:48:19.269 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-04-16T07:48:19.336 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-04-16T07:48:19.361 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-04-16T07:48:19.687 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-04-16T07:48:20.212 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 372 kB/s | 195 kB 00:00 2024-04-16T07:48:20.254 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 902 kB/s | 803 kB 00:00 2024-04-16T07:48:20.429 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-3 4.2 MB/s | 7.4 MB 00:01 2024-04-16T07:48:20.529 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 774 kB/s | 245 kB 00:00 2024-04-16T07:48:20.531 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:20.532 INFO:teuthology.orchestra.run.smithi118.stdout:Total 18 MB/s | 37 MB 00:02 2024-04-16T07:48:20.577 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:20.601 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:20.601 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:20.955 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:20.956 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:21.654 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:21.719 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T07:48:21.758 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T07:48:21.853 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:48:21.887 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T07:48:22.063 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:48:22.094 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:48:22.170 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T07:48:22.821 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T07:48:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T07:48:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T07:48:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T07:48:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:48:27.922 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 12/12 2024-04-16T07:48:27.922 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0a 1/12 2024-04-16T07:48:27.922 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T07:48:27.922 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T07:48:27.922 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T07:48:27.922 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T07:48:27.922 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T07:48:27.922 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T07:48:27.923 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T07:48:27.923 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T07:48:27.923 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T07:48:27.923 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T07:48:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T07:48:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:28.341 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:28.341 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T07:48:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T07:48:28.343 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:28.343 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:28.743 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2024-04-16T07:48:29.329 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:25 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:29.485 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 49 k 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T07:48:29.486 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2024-04-16T07:48:29.487 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:29.488 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.0 M 2024-04-16T07:48:29.488 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 26 M 2024-04-16T07:48:29.488 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:29.807 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 18 kB 00:00 2024-04-16T07:48:29.833 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 153 kB/s | 24 kB 00:00 2024-04-16T07:48:29.941 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 174 kB 00:00 2024-04-16T07:48:29.966 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 989 kB/s | 132 kB 00:00 2024-04-16T07:48:30.025 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 211 kB 00:00 2024-04-16T07:48:30.050 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 512 kB/s | 43 kB 00:00 2024-04-16T07:48:30.076 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): ceph-mgr-rook-19.0.0-3109.g30b0aa00.el9 122 kB/s | 49 kB 00:00 2024-04-16T07:48:30.484 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 35 kB/s | 15 kB 00:00 2024-04-16T07:48:30.535 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 63 kB/s | 32 kB 00:00 2024-04-16T07:48:30.760 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 263 kB/s | 59 kB 00:00 2024-04-16T07:48:30.785 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 286 kB/s | 203 kB 00:00 2024-04-16T07:48:30.985 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 398 kB/s | 90 kB 00:00 2024-04-16T07:48:31.119 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.6 MB/s | 1.0 MB 00:00 2024-04-16T07:48:31.123 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:31.123 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-04-16T07:48:31.123 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:31.137 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:31.137 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:31.229 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:31.230 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:31.755 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:31.923 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T07:48:31.978 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T07:48:32.033 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T07:48:32.063 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T07:48:32.104 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T07:48:32.199 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T07:48:32.307 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T07:48:32.364 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T07:48:33.089 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T07:48:33.136 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T07:48:33.170 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T07:48:33.219 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T07:48:33.240 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 13/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 1/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T07:48:34.128 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T07:48:34.129 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T07:48:34.486 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T07:48:34.486 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:34.486 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:34.486 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:34.487 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:34.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T07:48:35.343 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:31 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:35.497 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:35.498 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-04-16T07:48:35.498 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:48:35.498 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-04-16T07:48:35.498 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 142 k 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-04-16T07:48:35.499 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:35.500 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 6.8 M 2024-04-16T07:48:35.500 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 33 M 2024-04-16T07:48:35.500 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:36.238 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-04-16T07:48:36.264 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3109.g30b0aa00.e 486 kB/s | 142 kB 00:00 2024-04-16T07:48:36.289 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 781 kB/s | 248 kB 00:00 2024-04-16T07:48:36.731 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 131 kB/s | 58 kB 00:00 2024-04-16T07:48:36.756 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 109 kB/s | 51 kB 00:00 2024-04-16T07:48:37.015 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 706 kB/s | 548 kB 00:00 2024-04-16T07:48:37.018 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:37.018 INFO:teuthology.orchestra.run.smithi118.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-04-16T07:48:37.031 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:37.041 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:37.041 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:37.128 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:37.130 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:37.429 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:37.603 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T07:48:37.666 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T07:48:38.195 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T07:48:38.276 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T07:48:38.340 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T07:48:38.362 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:48:39.187 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 6/6 2024-04-16T07:48:39.187 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 1/6 2024-04-16T07:48:39.187 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T07:48:39.188 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T07:48:39.188 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T07:48:39.188 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T07:48:39.499 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T07:48:39.499 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:39.499 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:39.499 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:39.499 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T07:48:39.499 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T07:48:39.500 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T07:48:39.500 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T07:48:39.500 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T07:48:39.500 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:39.500 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:39.756 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2024-04-16T07:48:40.335 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:36 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:40.490 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:40.490 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:40.490 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 871 k 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-04-16T07:48:40.491 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:40.492 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 950 k 2024-04-16T07:48:40.492 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 4.0 M 2024-04-16T07:48:40.492 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:40.626 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 589 kB/s | 79 kB 00:00 2024-04-16T07:48:40.859 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-fuse-19.0.0-3109.g30b0aa00.el9.x86_ 2.3 MB/s | 871 kB 00:00 2024-04-16T07:48:40.860 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:40.861 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.5 MB/s | 950 kB 00:00 2024-04-16T07:48:40.861 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:40.883 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:40.883 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:40.956 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:40.956 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:41.134 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:41.269 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T07:48:41.336 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:48:42.086 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2/2 2024-04-16T07:48:42.086 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/2 2024-04-16T07:48:42.384 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T07:48:42.384 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:42.384 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:42.384 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T07:48:42.384 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:42.385 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:42.690 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-volume 2024-04-16T07:48:43.288 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:39 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:43.443 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:19.0.0-3109.g30b0aa00.el9 ceph-noarch 273 k 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-04-16T07:48:43.444 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:43.445 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 342 k 2024-04-16T07:48:43.445 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 1.5 M 2024-04-16T07:48:43.445 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:43.574 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 535 kB/s | 69 kB 00:00 2024-04-16T07:48:43.724 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-volume-19.0.0-3109.g30b0aa00.el9.no 980 kB/s | 273 kB 00:00 2024-04-16T07:48:43.725 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:43.726 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.2 MB/s | 342 kB 00:00 2024-04-16T07:48:43.726 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:43.734 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:43.734 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:43.996 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:44.178 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T07:48:44.219 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:48:44.235 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2/2 2024-04-16T07:48:44.236 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:48:44.236 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T07:48:44.236 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:45.004 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 1/2 2024-04-16T07:48:45.324 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T07:48:45.324 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:45.325 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:45.325 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:19.0.0-3109.g30b0aa00.el9.noarch 2024-04-16T07:48:45.325 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T07:48:45.325 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:45.325 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:45.632 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2024-04-16T07:48:46.219 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:42 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:46.380 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 125 k 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-04-16T07:48:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:46.382 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 125 k 2024-04-16T07:48:46.382 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 445 k 2024-04-16T07:48:46.382 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:46.624 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-19.0.0-3109.g30b0aa00.el9.x86_64 517 kB/s | 125 kB 00:00 2024-04-16T07:48:46.624 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:46.625 INFO:teuthology.orchestra.run.smithi118.stdout:Total 513 kB/s | 125 kB 00:00 2024-04-16T07:48:46.625 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:46.630 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:46.630 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:46.689 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:46.689 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:46.958 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:47.494 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:47.884 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:47.884 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:47.885 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:47.885 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:47.885 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:47.885 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:48.122 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2024-04-16T07:48:48.713 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:44 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:48.814 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:48.866 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:48.867 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:48.868 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:48.937 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2024-04-16T07:48:49.517 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:45 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:49.667 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:49.668 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-04-16T07:48:49.668 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:49.668 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 32 k 2024-04-16T07:48:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:49.668 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:49.669 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:49.669 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-04-16T07:48:49.669 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:49.669 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 32 k 2024-04-16T07:48:49.669 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 142 k 2024-04-16T07:48:49.669 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:49.837 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-19.0.0-3109.g30b0aa00.el9.x86_6 188 kB/s | 32 kB 00:00 2024-04-16T07:48:49.837 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:49.838 INFO:teuthology.orchestra.run.smithi118.stdout:Total 187 kB/s | 32 kB 00:00 2024-04-16T07:48:49.838 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:49.841 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:49.841 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:49.866 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:49.867 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:50.069 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:50.486 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:50.747 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:50.747 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:50.747 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:50.747 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:50.747 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:50.748 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:50.987 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2024-04-16T07:48:51.565 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:47 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:51.665 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:51.717 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:51.718 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:51.718 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:51.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2024-04-16T07:48:52.373 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:48 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:52.474 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:52.526 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:52.527 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:52.527 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:52.595 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2024-04-16T07:48:53.170 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:49 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:53.268 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:53.319 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:53.320 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:53.320 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:53.392 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2024-04-16T07:48:53.975 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:49 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:54.076 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:54.127 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:54.128 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:54.128 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:54.201 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2024-04-16T07:48:54.786 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:50 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:54.886 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:54.937 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:54.938 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:54.938 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:55.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2024-04-16T07:48:55.608 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:51 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:55.708 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 is already installed. 2024-04-16T07:48:55.759 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:55.761 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-04-16T07:48:55.761 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:55.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2024-04-16T07:48:56.422 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:52 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:56.576 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:56.576 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 90 k 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 90 k 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 313 k 2024-04-16T07:48:56.577 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:48:57.390 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 110 kB/s | 90 kB 00:00 2024-04-16T07:48:57.391 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:48:57.391 INFO:teuthology.orchestra.run.smithi118.stdout:Total 110 kB/s | 90 kB 00:00 2024-04-16T07:48:57.392 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:48:57.401 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:48:57.401 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:48:57.460 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:48:57.460 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:48:57.601 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:48:57.706 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:58.139 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:58.447 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:48:58.448 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:58.448 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:48:58.448 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:48:58.448 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:58.448 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:48:58.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2024-04-16T07:48:59.305 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:06:55 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:48:59.460 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:48:59.461 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:59.461 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-04-16T07:48:59.461 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:59.461 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:48:59.461 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 3.4 M 2024-04-16T07:48:59.461 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:59.461 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:48:59.462 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:48:59.462 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-04-16T07:48:59.462 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:48:59.462 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.4 M 2024-04-16T07:48:59.462 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 29 M 2024-04-16T07:48:59.462 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:49:02.311 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 1.2 MB/s | 3.4 MB 00:02 2024-04-16T07:49:02.312 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:49:02.312 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.2 MB/s | 3.4 MB 00:02 2024-04-16T07:49:02.312 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:49:02.325 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:49:02.325 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:49:02.402 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:49:02.403 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:49:02.970 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:49:03.062 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:03.089 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T07:49:03.089 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T07:49:03.089 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:49:03.089 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T07:49:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:04.362 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:04.362 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:04.363 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:49:04.363 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:49:04.363 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:04.363 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:49:04.608 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2024-04-16T07:49:05.203 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:07:01 ago on Tue 16 Apr 2024 07:42:04 AM UTC. 2024-04-16T07:49:05.356 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:19.0.0-3109.g30b0aa00.el9 ceph 175 k 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:05.357 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 175 k 2024-04-16T07:49:05.358 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 673 k 2024-04-16T07:49:05.358 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-04-16T07:49:05.725 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-19.0.0-3109.g30b0aa00.el9.x86_64.rpm 478 kB/s | 175 kB 00:00 2024-04-16T07:49:05.725 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-04-16T07:49:05.726 INFO:teuthology.orchestra.run.smithi118.stdout:Total 476 kB/s | 175 kB 00:00 2024-04-16T07:49:05.726 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-04-16T07:49:05.735 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-04-16T07:49:05.735 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-04-16T07:49:05.791 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-04-16T07:49:05.791 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-04-16T07:49:05.946 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-04-16T07:49:06.065 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:06.510 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:06.806 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 1/1 2024-04-16T07:49:06.806 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:06.806 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-04-16T07:49:06.807 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:19.0.0-3109.g30b0aa00.el9.x86_64 2024-04-16T07:49:06.807 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:06.807 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-04-16T07:49:07.109 DEBUG:teuthology.parallel:result is None 2024-04-16T07:49:07.109 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=30b0aa0009a925d6a46536e55c5656edcd51ecdb 2024-04-16T07:49:07.252 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T07:49:07.274 INFO:teuthology.orchestra.run.smithi118.stdout:19.0.0-3109.g30b0aa00.el9 2024-04-16T07:49:07.274 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3109.g30b0aa00.el9 2024-04-16T07:49:07.274 INFO:teuthology.task.install:The correct ceph version 19.0.0-3109.g30b0aa00 is installed. 2024-04-16T07:49:07.275 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T07:49:07.276 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:07.276 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T07:49:07.342 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T07:49:07.343 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:07.343 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T07:49:07.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T07:49:07.478 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T07:49:07.478 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:07.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T07:49:07.546 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T07:49:07.614 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T07:49:07.614 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:07.614 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T07:49:07.682 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T07:49:07.750 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T07:49:07.823 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T07:49:07.823 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2024-04-16T07:49:07.851 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T07:49:07.851 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T07:49:07.919 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T07:49:07.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T07:49:07.995 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T07:49:07.995 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T07:49:07.995 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987', 'branch': 'wip-matanb-crimson-replicated-april-15', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format}', 'email': None, 'first_in_suite': False, 'job_id': '7657987', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-replicated-april-15'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'a7639cb6ce633dabc4d37769d62caacb54582984'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6444, 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'a7639cb6ce633dabc4d37769d62caacb54582984', 'targets': {'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNb2mDReFOOnitsN7Mx+s7up9mh1GZXUj+FWZM1MBiYPxPvo1m4sNSkILYTUclrN6TJ6htpyremOhQ7ClHxDqfI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': '30b0aa0009a925d6a46536e55c5656edcd51ecdb', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_07:26:08', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T07:49:07.995 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T07:49:08.043 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T07:49:08.112 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:08.112 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T07:49:08.168 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T07:49:08.168 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-04-16T07:49:08.223 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T07:49:08.223 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:49:08.223 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-16T07:49:08.223 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:49:08.223 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:49:08.223 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-04-16 07:49:03.325850300 +0000 2024-04-16T07:49:08.224 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-04-16 07:41:06.906585735 +0000 2024-04-16T07:49:08.224 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-04-16 07:41:06.906585735 +0000 2024-04-16T07:49:08.224 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-04-16 07:41:06.906585735 +0000 2024-04-16T07:49:08.224 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T07:49:08.290 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-04-16T07:49:08.290 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-04-16T07:49:08.290 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.00010304 s, 5.0 MB/s 2024-04-16T07:49:08.291 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T07:49:08.349 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-04-16 07:49:03.326850278 +0000 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-04-16 07:41:07.258577955 +0000 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-04-16 07:41:07.258577955 +0000 2024-04-16T07:49:08.405 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-04-16 07:41:07.258577955 +0000 2024-04-16T07:49:08.405 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T07:49:08.471 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-04-16T07:49:08.471 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-04-16T07:49:08.471 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000100417 s, 5.1 MB/s 2024-04-16T07:49:08.472 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T07:49:08.528 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-04-16T07:49:08.584 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T07:49:08.584 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:49:08.584 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-16T07:49:08.584 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:49:08.584 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:49:08.584 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-04-16 07:49:03.326850278 +0000 2024-04-16T07:49:08.584 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-04-16 07:41:07.597570462 +0000 2024-04-16T07:49:08.584 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-04-16 07:41:07.597570462 +0000 2024-04-16T07:49:08.585 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-04-16 07:41:07.597570462 +0000 2024-04-16T07:49:08.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T07:49:08.651 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-04-16T07:49:08.651 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-04-16T07:49:08.651 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 9.49e-05 s, 5.4 MB/s 2024-04-16T07:49:08.652 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T07:49:08.709 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-04-16 07:49:03.326850278 +0000 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-04-16 07:41:07.930563101 +0000 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-04-16 07:41:07.930563101 +0000 2024-04-16T07:49:08.764 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-04-16 07:41:07.930563101 +0000 2024-04-16T07:49:08.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T07:49:08.830 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-04-16T07:49:08.830 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-04-16T07:49:08.830 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000106064 s, 4.8 MB/s 2024-04-16T07:49:08.831 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T07:49:08.887 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T07:49:08.887 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-16T07:49:08.887 INFO:tasks.ceph:Generating config... 2024-04-16T07:49:08.888 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-16T07:49:08.889 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-16T07:49:08.889 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-16T07:49:08.889 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-16T07:49:08.889 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-16T07:49:08.889 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T07:49:08.889 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T07:49:08.889 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T07:49:08.889 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-16T07:49:08.889 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-16T07:49:08.889 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-16T07:49:08.889 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-16T07:49:08.889 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T07:49:08.889 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T07:49:08.889 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-16T07:49:08.889 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:49:08.890 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-16T07:49:08.890 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-16T07:49:08.890 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-16T07:49:08.890 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-16T07:49:08.890 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-16T07:49:08.890 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-16T07:49:08.890 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T07:49:08.890 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T07:49:08.890 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-04-16T07:49:08.890 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-04-16T07:49:08.890 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-04-16T07:49:08.890 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T07:49:08.891 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T07:49:08.987 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T07:49:08.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T07:49:09.044 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:49:09.072 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.118')] 2024-04-16T07:49:09.073 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.118', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-04-16T07:49:09.073 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:09.073 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:49:09.128 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.118 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:09.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:09.218+0000 7fc5a6b22f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:09.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:09.218+0000 7fc5a6b22f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:09.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:09.218+0000 7fc5a6b22f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: generated fsid 45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:setting min_mon_release = quincy 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:epoch 0 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:fsid 45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:last_changed 2024-04-16T07:49:09.220008+0000 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:created 2024-04-16T07:49:09.220008+0000 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:min_mon_release 17 (quincy) 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:election_strategy: 1 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:0: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.a 2024-04-16T07:49:09.219 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-16T07:49:09.221 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T07:49:09.236 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00... 2024-04-16T07:49:09.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2024-04-16T07:49:09.328 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2024-04-16T07:49:09.329 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.330 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = 172.21.15.118 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: ms client mode = crc 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: ms cluster mode = crc 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: ms mon client mode = crc 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: ms mon cluster mode = crc 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: ms mon service mode = crc 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: ms service mode = crc 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2024-04-16T07:49:09.331 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug trim objects = true 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2024-04-16T07:49:09.332 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: debug monc = 20 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: crimson osd obc lru size = 10 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: osd shutdown pgref assert = True 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.333 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2024-04-16T07:49:09.334 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.335 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: mon min osdmap epochs = 50 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune interval = 2 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune min = 15 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune txsize = 2 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: paxos service trim min = 10 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default crimson = True 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2024-04-16T07:49:09.336 INFO:teuthology.orchestra.run.smithi118.stdout: crimson seastar cpu cores = 0-1 2024-04-16T07:49:09.337 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2024-04-16T07:49:09.337 INFO:teuthology.orchestra.run.smithi118.stdout: crimson seastar cpu cores = 2-3 2024-04-16T07:49:09.337 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2024-04-16T07:49:09.337 INFO:teuthology.orchestra.run.smithi118.stdout: crimson seastar cpu cores = 4-5 2024-04-16T07:49:09.342 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T07:49:09.342 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T07:49:09.409 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T07:49:09.409 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:09.409 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T07:49:09.425 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:09.425 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T07:49:09.479 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T07:49:09.480 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:09.480 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T07:49:09.480 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T07:49:09.561 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:09.561 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:09.617 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T07:49:09.617 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T07:49:09.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:49:09.722 INFO:teuthology.orchestra.run.smithi118.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T07:49:09.725 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T07:49:09.725 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T07:49:09.726 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T07:49:09.777 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T07:49:09.794 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T07:49:09.794 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-16T07:49:09.794 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T07:49:09.821 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T07:49:09.822 INFO:tasks.ceph:role: osd.0 2024-04-16T07:49:09.822 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T07:49:09.822 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T07:49:09.889 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:49:09.889 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:49:09.889 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:49:09.889 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:49:09.889 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:49:09.890 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:49:09.890 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:49:09.890 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:49:09.890 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:49:09.890 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:49:09.922 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-04-16T07:49:09.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-04-16T07:49:09.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T07:49:09.996 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T07:49:10.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T07:49:10.139 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T07:49:10.139 INFO:tasks.ceph:role: osd.1 2024-04-16T07:49:10.139 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T07:49:10.139 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:49:10.208 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:49:10.240 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-04-16T07:49:10.242 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-04-16T07:49:10.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T07:49:10.315 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T07:49:10.387 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T07:49:10.458 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T07:49:10.458 INFO:tasks.ceph:role: osd.2 2024-04-16T07:49:10.458 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T07:49:10.459 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T07:49:10.527 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T07:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T07:49:10.561 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-04-16T07:49:10.563 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-04-16T07:49:10.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-16T07:49:10.635 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T07:49:10.708 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:11.081 INFO:teuthology.orchestra.run.smithi118.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:11.082 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:11,084 seastar - Reactor backend: linux-aio 2024-04-16T07:49:11.098 INFO:teuthology.orchestra.run.smithi118.stderr:==29369==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:11.220 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:11,222 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:49:11.221 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:11,222 [shard 0:main] osd - get_early_config: set --cpuset 0-1 2024-04-16T07:49:11.221 INFO:teuthology.orchestra.run.smithi118.stderr:DEBUG 2024-04-16 07:49:11,223 [shard 0:main] seastar - reactor::drain 2024-04-16T07:49:11.793 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:11.793 INFO:teuthology.orchestra.run.smithi118.stderr:Suppressions used: 2024-04-16T07:49:11.793 INFO:teuthology.orchestra.run.smithi118.stderr: count bytes template 2024-04-16T07:49:11.793 INFO:teuthology.orchestra.run.smithi118.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:11.793 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:11.793 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-16T07:49:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:11.859 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:11,860 seastar - Reactor backend: linux-aio 2024-04-16T07:49:11.866 INFO:teuthology.orchestra.run.smithi118.stderr:==29367==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:15.291 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:15.292 INFO:teuthology.orchestra.run.smithi118.stderr:Suppressions used: 2024-04-16T07:49:15.292 INFO:teuthology.orchestra.run.smithi118.stderr: count bytes template 2024-04-16T07:49:15.292 INFO:teuthology.orchestra.run.smithi118.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:15.292 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:15.292 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-16T07:49:15.292 INFO:teuthology.orchestra.run.smithi118.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00 2024-04-16T07:49:15.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T07:49:15.345 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:15.640 INFO:teuthology.orchestra.run.smithi118.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:15.641 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:15,643 seastar - Reactor backend: linux-aio 2024-04-16T07:49:15.646 INFO:teuthology.orchestra.run.smithi118.stderr:==30176==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:15.765 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:15,766 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:49:15.765 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:15,766 [shard 0:main] osd - get_early_config: set --cpuset 2-3 2024-04-16T07:49:15.766 INFO:teuthology.orchestra.run.smithi118.stderr:DEBUG 2024-04-16 07:49:15,767 [shard 0:main] seastar - reactor::drain 2024-04-16T07:49:16.322 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:16.323 INFO:teuthology.orchestra.run.smithi118.stderr:Suppressions used: 2024-04-16T07:49:16.323 INFO:teuthology.orchestra.run.smithi118.stderr: count bytes template 2024-04-16T07:49:16.323 INFO:teuthology.orchestra.run.smithi118.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:16.323 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:16.323 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-16T07:49:16.402 INFO:teuthology.orchestra.run.smithi118.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:16.406 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:16,407 seastar - Reactor backend: linux-aio 2024-04-16T07:49:16.413 INFO:teuthology.orchestra.run.smithi118.stderr:==30174==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:16.478 INFO:teuthology.orchestra.run.smithi118.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99d759 0xb870220 0xb87179c 0xb8719b2 0xb871b08 0xb871fd1 0x54daf 0xc570c 0x6854b 0x59da06b 0x5a2bc29 0x5e0d781 0x5d8c2f0 0x5d8c4ca 0x5d8ca56 0x5bd1519 0x5bec5dd 0x6ec6e84 0x6ed8657 0x6ed9af4 0x6ed9b79 0x6ed9c7b 0x6ed9d73 0x6ed9f80 0x6eda166 0x6eda4b5 0x6eda859 0x6edaa89 0x6edacf3 0x6edafc0 0x6edb1de 0x6edb296 0x6edb35c 0x6edb43d 0x5a39e18 0x5a39f22 0x5a39fde 0x5a3a09a 0x5a3a156 0x5a3a2a3 0x5a3a42e 0x5a3a7ef 0x5a56730 0xb882ea5 0xb89d13f 0xb93e13d 0xb93f78b 0xb61bf71 0xb61c8ed 0x367a090 0x3feaf 0x3ff5f 0x3463444 2024-04-16T07:49:16.478 INFO:teuthology.orchestra.run.smithi118.stderr:kernel callstack: 2024-04-16T07:49:19.856 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:19.856 INFO:teuthology.orchestra.run.smithi118.stderr:Suppressions used: 2024-04-16T07:49:19.856 INFO:teuthology.orchestra.run.smithi118.stderr: count bytes template 2024-04-16T07:49:19.856 INFO:teuthology.orchestra.run.smithi118.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:19.856 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:19.857 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-16T07:49:19.857 INFO:teuthology.orchestra.run.smithi118.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00 2024-04-16T07:49:19.880 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T07:49:19.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:20.201 INFO:teuthology.orchestra.run.smithi118.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:20.202 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:20,204 seastar - Reactor backend: linux-aio 2024-04-16T07:49:20.207 INFO:teuthology.orchestra.run.smithi118.stderr:==30982==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:20,326 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T07:49:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:20,326 [shard 0:main] osd - get_early_config: set --cpuset 4-5 2024-04-16T07:49:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:DEBUG 2024-04-16 07:49:20,327 [shard 0:main] seastar - reactor::drain 2024-04-16T07:49:20.865 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:20.865 INFO:teuthology.orchestra.run.smithi118.stderr:Suppressions used: 2024-04-16T07:49:20.865 INFO:teuthology.orchestra.run.smithi118.stderr: count bytes template 2024-04-16T07:49:20.865 INFO:teuthology.orchestra.run.smithi118.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:20.866 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:20.866 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-16T07:49:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T07:49:20.926 INFO:teuthology.orchestra.run.smithi118.stderr:INFO 2024-04-16 07:49:20,927 seastar - Reactor backend: linux-aio 2024-04-16T07:49:20.939 INFO:teuthology.orchestra.run.smithi118.stderr:==30980==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T07:49:24.362 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:24.362 INFO:teuthology.orchestra.run.smithi118.stderr:Suppressions used: 2024-04-16T07:49:24.362 INFO:teuthology.orchestra.run.smithi118.stderr: count bytes template 2024-04-16T07:49:24.362 INFO:teuthology.orchestra.run.smithi118.stderr: 1 8 MallocExtension::Register 2024-04-16T07:49:24.362 INFO:teuthology.orchestra.run.smithi118.stderr:----------------------------------------------------- 2024-04-16T07:49:24.362 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-16T07:49:24.362 INFO:teuthology.orchestra.run.smithi118.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00 2024-04-16T07:49:24.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T07:49:24.414 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T07:49:24.414 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:24.414 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T07:49:24.483 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:24.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T07:49:24.551 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:24.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T07:49:24.619 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:24.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T07:49:24.687 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:24.687 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T07:49:24.742 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T07:49:24.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQB1LR5m1msoKxAAt6xjfI5ltvM5FSOFd3I4ig== 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQB4LR5mqNlrABAA8abAG3Ys8EvdwvF8anCePg== 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQB8LR5m6yJEIRAAtmF/iwivwBQ7R0Rmzt3aWg== 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCBLR5mVDK3BBAA0Htu5lYndglfqhxzPLdyLA== 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-04-16T07:49:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQB1LR5mWNxuLhAABM5G497WR/lRgJaHkYGomw== 2024-04-16T07:49:24.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T07:49:24.913 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:49:24.977 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:49:25.040 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T07:49:25.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T07:49:25.171 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T07:49:25.171 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T07:49:25.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T07:49:25.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:25.325+0000 7f0ee4073a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:25.325+0000 7f0ee4073a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T07:49:25.366 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T07:49:25.422 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T07:49:25.422 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T07:49:25.422 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T07:49:25.464 INFO:tasks.ceph.mon.a:Started 2024-04-16T07:49:25.464 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T07:49:25.464 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T07:49:25.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T07:49:25.466 INFO:tasks.ceph.mgr.x:Started 2024-04-16T07:49:25.466 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T07:49:25.466 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T07:49:25.550 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:25.550+0000 7fefb1a36000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.551 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:49:25.550+0000 7f56ac677a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.551 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:49:25.550+0000 7f56ac677a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.551 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:49:25.551+0000 7f56ac677a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:25.574+0000 7f460986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:25.585+0000 7f460986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.654 INFO:teuthology.orchestra.run.smithi118.stderr:adjusted tunables profile to default 2024-04-16T07:49:25.665 INFO:tasks.ceph:check_enable_crimson: True 2024-04-16T07:49:25.666 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-16T07:49:25.666 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-16T07:49:25.754 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:25.755+0000 7fefb1a36000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.755 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:25.755+0000 7fefb1a36000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:25.772+0000 7f70bfe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:25.780+0000 7f70bfe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:25.862 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:49:25.863+0000 7f56a33f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-16T07:49:25.862 INFO:tasks.ceph.mon.a.smithi118.stderr:Please be aware that this feature is experimental, untested, 2024-04-16T07:49:25.862 INFO:tasks.ceph.mon.a.smithi118.stderr:unsupported, and may result in data corruption, data loss, 2024-04-16T07:49:25.862 INFO:tasks.ceph.mon.a.smithi118.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-16T07:49:25.863 INFO:tasks.ceph.mon.a.smithi118.stderr:feature with important data. 2024-04-16T07:49:25.863 INFO:tasks.ceph.mon.a.smithi118.stderr: 2024-04-16T07:49:26.267 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:26.267+0000 7fefb1a36000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T07:49:26.321 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:26.321+0000 7fefb1a36000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T07:49:26.666 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T07:49:26.667 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:26.667 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T07:49:26.697 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:26.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T07:49:26.765 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:49:26.765 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T07:49:26.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new e9e83689-d273-47a8-9071-cdc9bd8da224 0 2024-04-16T07:49:26.886 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:26.886+0000 7fefb1a36000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T07:49:26.943 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:26.944+0000 7fefb1a36000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T07:49:26.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:26.982+0000 7f4eb159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:26.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:26.991+0000 7f4eb159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:26.991 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:26.992+0000 7fefb1a36000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T07:49:27.061 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:27.061+0000 7fefb1a36000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T07:49:27.088 INFO:teuthology.orchestra.run.smithi118.stdout:0 2024-04-16T07:49:27.100 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new 6a5b47f2-0876-4901-a565-f9bb9968ac9c 1 2024-04-16T07:49:27.107 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:27.107+0000 7fefb1a36000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T07:49:27.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:27.213+0000 7f9653b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:27.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:27.222+0000 7f9653b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:27.236 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:27.236+0000 7fefb1a36000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T07:49:27.322 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:27.323+0000 7fefb1a36000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T07:49:27.325 INFO:teuthology.orchestra.run.smithi118.stdout:1 2024-04-16T07:49:27.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd new c218e742-6d70-478b-939d-c08b24ba6c4f 2 2024-04-16T07:49:27.373 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:27.373+0000 7fefb1a36000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T07:49:27.429 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:27.430+0000 7fefb1a36000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T07:49:27.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:27.443+0000 7fa24b4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:27.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:27.452+0000 7fa24b4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:27.480 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:27.480+0000 7fefb1a36000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T07:49:27.532 INFO:teuthology.orchestra.run.smithi118.stdout:2 2024-04-16T07:49:27.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-16T07:49:27.617 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T07:49:27.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T07:49:27.660 INFO:tasks.ceph.osd.0:Started 2024-04-16T07:49:27.660 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-16T07:49:27.692 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T07:49:27.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T07:49:27.734 INFO:tasks.ceph.osd.1:Started 2024-04-16T07:49:27.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-16T07:49:27.767 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T07:49:27.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T07:49:27.810 INFO:tasks.ceph.osd.2:Started 2024-04-16T07:49:27.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:27.938 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:27.939+0000 7fefb1a36000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T07:49:27.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:27.983+0000 7fe5076ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:27.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:27.995+0000 7fe5076ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:28.084 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":6,"fsid":"45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00","created":"2024-04-16T07:49:25.572071+0000","modified":"2024-04-16T07:49:27.533138+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:49:27.533138+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e9e83689-d273-47a8-9071-cdc9bd8da224","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6a5b47f2-0876-4901-a565-f9bb9968ac9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c218e742-6d70-478b-939d-c08b24ba6c4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:28.097 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T07:49:28.097 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T07:49:28.109 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:28.109+0000 7fefb1a36000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T07:49:28.235 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:28.235+0000 7fefb1a36000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T07:49:28.285 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:28.285+0000 7fefb1a36000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T07:49:28.400 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:28.407 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:28.407+0000 7fefb1a36000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:49:28.504 INFO:teuthology.misc.health.smithi118.stderr:2024-04-16T07:49:28.500+0000 7f57d20ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.507 INFO:teuthology.misc.health.smithi118.stderr:2024-04-16T07:49:28.508+0000 7f57d20ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:28.516 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:28.516+0000 7fefb1a36000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T07:49:28.576 INFO:teuthology.misc.health.smithi118.stdout: 2024-04-16T07:49:28.576 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":6,"fsid":"45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00","created":"2024-04-16T07:49:25.572071+0000","modified":"2024-04-16T07:49:27.533138+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T07:49:27.533138+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"e9e83689-d273-47a8-9071-cdc9bd8da224","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6a5b47f2-0876-4901-a565-f9bb9968ac9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c218e742-6d70-478b-939d-c08b24ba6c4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:28.588 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-16T07:49:28.688 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:28.688+0000 7fefb1a36000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T07:49:28.734 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:28.735+0000 7fefb1a36000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T07:49:28.904 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:28.904+0000 7fefb1a36000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T07:49:29.805 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:29.805+0000 7fefb1a36000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T07:49:30.101 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:30.101+0000 7fefb1a36000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T07:49:30.162 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:30.162+0000 7fefb1a36000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T07:49:30.295 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:30.295+0000 7fefb1a36000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T07:49:31.304 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T07:49:31.304+0000 7fef50711640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T07:49:34.890 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:35.001 INFO:teuthology.misc.health.smithi118.stderr:2024-04-16T07:49:34.997+0000 7f278191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:35.005 INFO:teuthology.misc.health.smithi118.stderr:2024-04-16T07:49:35.005+0000 7f278191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:35.193 INFO:teuthology.misc.health.smithi118.stdout: 2024-04-16T07:49:35.193 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":9,"fsid":"45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00","created":"2024-04-16T07:49:25.572071+0000","modified":"2024-04-16T07:49:34.359579+0000","last_up_change":"2024-04-16T07:49:32.353561+0000","last_in_change":"2024-04-16T07:49:27.533138+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:49:34.324725+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e9e83689-d273-47a8-9071-cdc9bd8da224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1815199286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":1815199286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1815199286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":1815199286}]},"public_addr":"172.21.15.118:6802/1815199286","cluster_addr":"172.21.15.118:6803/1815199286","heartbeat_back_addr":"172.21.15.118:6812/1815199286","heartbeat_front_addr":"172.21.15.118:6813/1815199286","state":["exists","up"]},{"osd":1,"uuid":"6a5b47f2-0876-4901-a565-f9bb9968ac9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2962547363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2962547363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2962547363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":2962547363}]},"public_addr":"172.21.15.118:6806/2962547363","cluster_addr":"172.21.15.118:6805/2962547363","heartbeat_back_addr":"172.21.15.118:6804/2962547363","heartbeat_front_addr":"172.21.15.118:6809/2962547363","state":["exists","up"]},{"osd":2,"uuid":"c218e742-6d70-478b-939d-c08b24ba6c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":866120993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":866120993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":866120993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":866120993}]},"public_addr":"172.21.15.118:6808/866120993","cluster_addr":"172.21.15.118:6807/866120993","heartbeat_back_addr":"172.21.15.118:6811/866120993","heartbeat_front_addr":"172.21.15.118:6810/866120993","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:35.205 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-16T07:49:35.205 INFO:tasks.ceph:Creating RBD pool 2024-04-16T07:49:35.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-16T07:49:35.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:35.310+0000 7fced6606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:35.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:35.320+0000 7fced6606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:36.373 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'rbd' created 2024-04-16T07:49:36.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T07:49:36.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:36.531+0000 7f2f5926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:36.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:36.544+0000 7f2f5926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:37.377 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T07:49:37.400 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T07:49:37.400 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T07:49:37.400 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T07:49:37.400 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T07:49:37.401 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T07:49:37.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:37.542+0000 7f3b6746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:37.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:37.549+0000 7f3b6746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:37.785 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:37.798 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":644684926},{"type":"v1","addr":"172.21.15.118:6801","nonce":644684926}]},"active_addr":"172.21.15.118:6801/644684926","active_change":"2024-04-16T07:49:30.298167+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":2346866520}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":1856096891}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":3683085655}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":2892119104}]}]} 2024-04-16T07:49:37.800 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T07:49:37.800 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T07:49:37.800 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:37.910+0000 7f77b946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:37.917+0000 7f77b946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.103 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:38.103 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":12,"fsid":"45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00","created":"2024-04-16T07:49:25.572071+0000","modified":"2024-04-16T07:49:37.374564+0000","last_up_change":"2024-04-16T07:49:32.353561+0000","last_in_change":"2024-04-16T07:49:27.533138+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:49:34.324725+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:49:35.527658+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e9e83689-d273-47a8-9071-cdc9bd8da224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1815199286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":1815199286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1815199286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":1815199286}]},"public_addr":"172.21.15.118:6802/1815199286","cluster_addr":"172.21.15.118:6803/1815199286","heartbeat_back_addr":"172.21.15.118:6812/1815199286","heartbeat_front_addr":"172.21.15.118:6813/1815199286","state":["exists","up"]},{"osd":1,"uuid":"6a5b47f2-0876-4901-a565-f9bb9968ac9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2962547363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2962547363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2962547363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":2962547363}]},"public_addr":"172.21.15.118:6806/2962547363","cluster_addr":"172.21.15.118:6805/2962547363","heartbeat_back_addr":"172.21.15.118:6804/2962547363","heartbeat_front_addr":"172.21.15.118:6809/2962547363","state":["exists","up"]},{"osd":2,"uuid":"c218e742-6d70-478b-939d-c08b24ba6c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":866120993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":866120993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":866120993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":866120993}]},"public_addr":"172.21.15.118:6808/866120993","cluster_addr":"172.21.15.118:6807/866120993","heartbeat_back_addr":"172.21.15.118:6811/866120993","heartbeat_front_addr":"172.21.15.118:6810/866120993","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:38.116 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T07:49:38.116 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T07:49:38.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.225+0000 7f55a1c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.232+0000 7f55a1c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.432 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:38.432 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":13,"fsid":"45b83fcd-bf3e-4f53-b5ec-e3fb9ba86b00","created":"2024-04-16T07:49:25.572071+0000","modified":"2024-04-16T07:49:38.401586+0000","last_up_change":"2024-04-16T07:49:32.353561+0000","last_in_change":"2024-04-16T07:49:27.533138+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T07:49:34.324725+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T07:49:35.527658+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e9e83689-d273-47a8-9071-cdc9bd8da224","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1815199286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":1815199286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1815199286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":1815199286}]},"public_addr":"172.21.15.118:6802/1815199286","cluster_addr":"172.21.15.118:6803/1815199286","heartbeat_back_addr":"172.21.15.118:6812/1815199286","heartbeat_front_addr":"172.21.15.118:6813/1815199286","state":["exists","up"]},{"osd":1,"uuid":"6a5b47f2-0876-4901-a565-f9bb9968ac9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2962547363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2962547363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2962547363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":2962547363}]},"public_addr":"172.21.15.118:6806/2962547363","cluster_addr":"172.21.15.118:6805/2962547363","heartbeat_back_addr":"172.21.15.118:6804/2962547363","heartbeat_front_addr":"172.21.15.118:6809/2962547363","state":["exists","up"]},{"osd":2,"uuid":"c218e742-6d70-478b-939d-c08b24ba6c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":866120993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":866120993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":866120993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":866120993}]},"public_addr":"172.21.15.118:6808/866120993","cluster_addr":"172.21.15.118:6807/866120993","heartbeat_back_addr":"172.21.15.118:6811/866120993","heartbeat_front_addr":"172.21.15.118:6810/866120993","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T07:49:38.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T07:49:38.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T07:49:38.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T07:49:38.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.561+0000 7f25aba64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.561+0000 7f6553a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.561+0000 7f36704ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.570+0000 7f36704ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.570+0000 7f6553a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.571+0000 7f25aba64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.603 INFO:teuthology.orchestra.run.smithi118.stdout:34359738374 2024-04-16T07:49:38.604 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:49:38.606 INFO:teuthology.orchestra.run.smithi118.stdout:34359738374 2024-04-16T07:49:38.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:49:38.608 INFO:teuthology.orchestra.run.smithi118.stdout:34359738374 2024-04-16T07:49:38.608 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:49:38.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.715+0000 7f07e2224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.724+0000 7f07e2224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.761+0000 7f75fa813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.769+0000 7f75fa813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.773+0000 7fa943b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:38.783+0000 7fa943b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:38.938 INFO:teuthology.orchestra.run.smithi118.stdout:34359738373 2024-04-16T07:49:38.951 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.1 2024-04-16T07:49:38.969 INFO:teuthology.orchestra.run.smithi118.stdout:34359738373 2024-04-16T07:49:38.976 INFO:teuthology.orchestra.run.smithi118.stdout:34359738373 2024-04-16T07:49:38.983 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.2 2024-04-16T07:49:38.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2024-04-16T07:49:39.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:49:39.984 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:49:39.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:49:40.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:40.087+0000 7fa26b35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:40.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:40.098+0000 7fa26b35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:40.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:40.123+0000 7f3ff735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:40.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:40.132+0000 7f3ff735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:40.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:40.128+0000 7f52e586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:40.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:40.143+0000 7f52e586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:40.302 INFO:teuthology.orchestra.run.smithi118.stdout:34359738373 2024-04-16T07:49:40.317 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.1 2024-04-16T07:49:40.327 INFO:teuthology.orchestra.run.smithi118.stdout:34359738373 2024-04-16T07:49:40.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2024-04-16T07:49:40.341 INFO:teuthology.orchestra.run.smithi118.stdout:34359738373 2024-04-16T07:49:40.353 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.2 2024-04-16T07:49:41.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T07:49:41.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T07:49:41.390 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T07:49:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:41.510+0000 7f0e7455d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:41.510+0000 7fe39c55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:41.510+0000 7f293e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:41.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:41.519+0000 7f0e7455d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:41.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:41.519+0000 7f293e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:41.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:41.519+0000 7fe39c55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:41.725 INFO:teuthology.orchestra.run.smithi118.stdout:34359738374 2024-04-16T07:49:41.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2024-04-16T07:49:41.739 DEBUG:teuthology.parallel:result is None 2024-04-16T07:49:41.755 INFO:teuthology.orchestra.run.smithi118.stdout:34359738374 2024-04-16T07:49:41.760 INFO:teuthology.orchestra.run.smithi118.stdout:34359738374 2024-04-16T07:49:41.769 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2024-04-16T07:49:41.769 DEBUG:teuthology.parallel:result is None 2024-04-16T07:49:41.774 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2024-04-16T07:49:41.774 DEBUG:teuthology.parallel:result is None 2024-04-16T07:49:41.774 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T07:49:41.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:41.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:41.921+0000 7f4826eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:41.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:41.928+0000 7f4826eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:42.111 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:42.111 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-04-16T07:49:42.123 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-16T07:49:40.304150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82692,"kb_used_data":1460,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490108,"statfs":{"total":322122547200,"available":322037870592,"internally_reserved":0,"allocated":1495040,"data_stored":1251956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.904296"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.421193+0000","last_change":"2024-04-16T07:49:37.471331+0000","last_active":"2024-04-16T07:49:38.421193+0000","last_peered":"2024-04-16T07:49:38.421193+0000","last_clean":"2024-04-16T07:49:38.421193+0000","last_became_active":"2024-04-16T07:49:37.461606+0000","last_became_peered":"2024-04-16T07:49:37.461606+0000","last_unstale":"2024-04-16T07:49:38.421193+0000","last_undegraded":"2024-04-16T07:49:38.421193+0000","last_fullsized":"2024-04-16T07:49:38.421193+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.426284+0000","last_change":"2024-04-16T07:49:37.485553+0000","last_active":"2024-04-16T07:49:38.426284+0000","last_peered":"2024-04-16T07:49:38.426284+0000","last_clean":"2024-04-16T07:49:38.426284+0000","last_became_active":"2024-04-16T07:49:37.476587+0000","last_became_peered":"2024-04-16T07:49:37.476587+0000","last_unstale":"2024-04-16T07:49:38.426284+0000","last_undegraded":"2024-04-16T07:49:38.426284+0000","last_fullsized":"2024-04-16T07:49:38.426284+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.413613+0000","last_change":"2024-04-16T07:49:37.465405+0000","last_active":"2024-04-16T07:49:38.413613+0000","last_peered":"2024-04-16T07:49:38.413613+0000","last_clean":"2024-04-16T07:49:38.413613+0000","last_became_active":"2024-04-16T07:49:37.456323+0000","last_became_peered":"2024-04-16T07:49:37.456323+0000","last_unstale":"2024-04-16T07:49:38.413613+0000","last_undegraded":"2024-04-16T07:49:38.413613+0000","last_fullsized":"2024-04-16T07:49:38.413613+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.427950+0000","last_change":"2024-04-16T07:49:37.488274+0000","last_active":"2024-04-16T07:49:38.427950+0000","last_peered":"2024-04-16T07:49:38.427950+0000","last_clean":"2024-04-16T07:49:38.427950+0000","last_became_active":"2024-04-16T07:49:37.480935+0000","last_became_peered":"2024-04-16T07:49:37.480935+0000","last_unstale":"2024-04-16T07:49:38.427950+0000","last_undegraded":"2024-04-16T07:49:38.427950+0000","last_fullsized":"2024-04-16T07:49:38.427950+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:49:37.488727+0000","last_change":"2024-04-16T07:49:37.490201+0000","last_active":"2024-04-16T07:49:37.488727+0000","last_peered":"2024-04-16T07:49:37.488727+0000","last_clean":"2024-04-16T07:49:36.368828+0000","last_became_active":"2024-04-16T07:49:37.482113+0000","last_became_peered":"2024-04-16T07:49:37.482113+0000","last_unstale":"2024-04-16T07:49:37.488727+0000","last_undegraded":"2024-04-16T07:49:37.488727+0000","last_fullsized":"2024-04-16T07:49:37.488727+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:49:37.488452+0000","last_change":"2024-04-16T07:49:37.489953+0000","last_active":"2024-04-16T07:49:37.488452+0000","last_peered":"2024-04-16T07:49:37.488452+0000","last_clean":"2024-04-16T07:49:36.368828+0000","last_became_active":"2024-04-16T07:49:37.477919+0000","last_became_peered":"2024-04-16T07:49:37.477919+0000","last_unstale":"2024-04-16T07:49:37.488452+0000","last_undegraded":"2024-04-16T07:49:37.488452+0000","last_fullsized":"2024-04-16T07:49:37.488452+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:49:37.494215+0000","last_change":"2024-04-16T07:49:37.496034+0000","last_active":"2024-04-16T07:49:37.494215+0000","last_peered":"2024-04-16T07:49:37.494215+0000","last_clean":"2024-04-16T07:49:36.368828+0000","last_became_active":"2024-04-16T07:49:37.488554+0000","last_became_peered":"2024-04-16T07:49:37.488554+0000","last_unstale":"2024-04-16T07:49:37.494215+0000","last_undegraded":"2024-04-16T07:49:37.494215+0000","last_fullsized":"2024-04-16T07:49:37.494215+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.427070+0000","last_change":"2024-04-16T07:49:37.456628+0000","last_active":"2024-04-16T07:49:38.427070+0000","last_peered":"2024-04-16T07:49:38.427070+0000","last_clean":"2024-04-16T07:49:38.427070+0000","last_became_active":"2024-04-16T07:49:37.452448+0000","last_became_peered":"2024-04-16T07:49:37.452448+0000","last_unstale":"2024-04-16T07:49:38.427070+0000","last_undegraded":"2024-04-16T07:49:38.427070+0000","last_fullsized":"2024-04-16T07:49:38.427070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.417624+0000","last_change":"2024-04-16T07:49:35.395227+0000","last_active":"2024-04-16T07:49:38.417624+0000","last_peered":"2024-04-16T07:49:38.417624+0000","last_clean":"2024-04-16T07:49:38.417624+0000","last_became_active":"2024-04-16T07:49:35.390901+0000","last_became_peered":"2024-04-16T07:49:35.390901+0000","last_unstale":"2024-04-16T07:49:38.417624+0000","last_undegraded":"2024-04-16T07:49:38.417624+0000","last_fullsized":"2024-04-16T07:49:38.417624+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:34.359579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:34.359579+0000","last_clean_scrub_stamp":"2024-04-16T07:49:34.359579+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27052,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830548,"statfs":{"total":107374182400,"available":107346481152,"internally_reserved":0,"allocated":102400,"data_stored":23740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:42.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T07:49:42.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:42.232+0000 7fcfc186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:42.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:42.239+0000 7fcfc186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:42.427 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:42.428 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-04-16T07:49:42.440 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-16T07:49:42.304536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82692,"kb_used_data":1460,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490108,"statfs":{"total":322122547200,"available":322037870592,"internally_reserved":0,"allocated":1495040,"data_stored":1251956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.904682"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.421193+0000","last_change":"2024-04-16T07:49:37.471331+0000","last_active":"2024-04-16T07:49:38.421193+0000","last_peered":"2024-04-16T07:49:38.421193+0000","last_clean":"2024-04-16T07:49:38.421193+0000","last_became_active":"2024-04-16T07:49:37.461606+0000","last_became_peered":"2024-04-16T07:49:37.461606+0000","last_unstale":"2024-04-16T07:49:38.421193+0000","last_undegraded":"2024-04-16T07:49:38.421193+0000","last_fullsized":"2024-04-16T07:49:38.421193+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.426284+0000","last_change":"2024-04-16T07:49:37.485553+0000","last_active":"2024-04-16T07:49:38.426284+0000","last_peered":"2024-04-16T07:49:38.426284+0000","last_clean":"2024-04-16T07:49:38.426284+0000","last_became_active":"2024-04-16T07:49:37.476587+0000","last_became_peered":"2024-04-16T07:49:37.476587+0000","last_unstale":"2024-04-16T07:49:38.426284+0000","last_undegraded":"2024-04-16T07:49:38.426284+0000","last_fullsized":"2024-04-16T07:49:38.426284+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.413613+0000","last_change":"2024-04-16T07:49:37.465405+0000","last_active":"2024-04-16T07:49:38.413613+0000","last_peered":"2024-04-16T07:49:38.413613+0000","last_clean":"2024-04-16T07:49:38.413613+0000","last_became_active":"2024-04-16T07:49:37.456323+0000","last_became_peered":"2024-04-16T07:49:37.456323+0000","last_unstale":"2024-04-16T07:49:38.413613+0000","last_undegraded":"2024-04-16T07:49:38.413613+0000","last_fullsized":"2024-04-16T07:49:38.413613+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.427950+0000","last_change":"2024-04-16T07:49:37.488274+0000","last_active":"2024-04-16T07:49:38.427950+0000","last_peered":"2024-04-16T07:49:38.427950+0000","last_clean":"2024-04-16T07:49:38.427950+0000","last_became_active":"2024-04-16T07:49:37.480935+0000","last_became_peered":"2024-04-16T07:49:37.480935+0000","last_unstale":"2024-04-16T07:49:38.427950+0000","last_undegraded":"2024-04-16T07:49:38.427950+0000","last_fullsized":"2024-04-16T07:49:38.427950+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:49:37.488727+0000","last_change":"2024-04-16T07:49:37.490201+0000","last_active":"2024-04-16T07:49:37.488727+0000","last_peered":"2024-04-16T07:49:37.488727+0000","last_clean":"2024-04-16T07:49:36.368828+0000","last_became_active":"2024-04-16T07:49:37.482113+0000","last_became_peered":"2024-04-16T07:49:37.482113+0000","last_unstale":"2024-04-16T07:49:37.488727+0000","last_undegraded":"2024-04-16T07:49:37.488727+0000","last_fullsized":"2024-04-16T07:49:37.488727+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:49:37.488452+0000","last_change":"2024-04-16T07:49:37.489953+0000","last_active":"2024-04-16T07:49:37.488452+0000","last_peered":"2024-04-16T07:49:37.488452+0000","last_clean":"2024-04-16T07:49:36.368828+0000","last_became_active":"2024-04-16T07:49:37.477919+0000","last_became_peered":"2024-04-16T07:49:37.477919+0000","last_unstale":"2024-04-16T07:49:37.488452+0000","last_undegraded":"2024-04-16T07:49:37.488452+0000","last_fullsized":"2024-04-16T07:49:37.488452+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T07:49:37.494215+0000","last_change":"2024-04-16T07:49:37.496034+0000","last_active":"2024-04-16T07:49:37.494215+0000","last_peered":"2024-04-16T07:49:37.494215+0000","last_clean":"2024-04-16T07:49:36.368828+0000","last_became_active":"2024-04-16T07:49:37.488554+0000","last_became_peered":"2024-04-16T07:49:37.488554+0000","last_unstale":"2024-04-16T07:49:37.494215+0000","last_undegraded":"2024-04-16T07:49:37.494215+0000","last_fullsized":"2024-04-16T07:49:37.494215+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.427070+0000","last_change":"2024-04-16T07:49:37.456628+0000","last_active":"2024-04-16T07:49:38.427070+0000","last_peered":"2024-04-16T07:49:38.427070+0000","last_clean":"2024-04-16T07:49:38.427070+0000","last_became_active":"2024-04-16T07:49:37.452448+0000","last_became_peered":"2024-04-16T07:49:37.452448+0000","last_unstale":"2024-04-16T07:49:38.427070+0000","last_undegraded":"2024-04-16T07:49:38.427070+0000","last_fullsized":"2024-04-16T07:49:38.427070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:36.368828+0000","last_clean_scrub_stamp":"2024-04-16T07:49:36.368828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T07:49:38.417624+0000","last_change":"2024-04-16T07:49:35.395227+0000","last_active":"2024-04-16T07:49:38.417624+0000","last_peered":"2024-04-16T07:49:38.417624+0000","last_clean":"2024-04-16T07:49:38.417624+0000","last_became_active":"2024-04-16T07:49:35.390901+0000","last_became_peered":"2024-04-16T07:49:35.390901+0000","last_unstale":"2024-04-16T07:49:38.417624+0000","last_undegraded":"2024-04-16T07:49:38.417624+0000","last_fullsized":"2024-04-16T07:49:38.417624+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T07:49:34.359579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T07:49:34.359579+0000","last_clean_scrub_stamp":"2024-04-16T07:49:34.359579+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27052,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830548,"statfs":{"total":107374182400,"available":107346481152,"internally_reserved":0,"allocated":102400,"data_stored":23740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T07:49:42.441 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T07:49:42.441 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T07:49:42.441 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T07:49:42.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T07:49:42.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:42.549+0000 7fa9e8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:42.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T07:49:42.555+0000 7fa9e8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T07:49:42.779 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-04-16T07:49:42.779 INFO:teuthology.orchestra.run.smithi118.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T07:49:42.793 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T07:49:42.793 INFO:teuthology.run_tasks:Running task workunit... 2024-04-16T07:49:42.805 INFO:tasks.workunit:Pulling workunits from ref a7639cb6ce633dabc4d37769d62caacb54582984 2024-04-16T07:49:42.806 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-16T07:49:42.806 DEBUG:teuthology.orchestra.run.smithi118:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-16T07:49:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T07:49:42.821 INFO:teuthology.orchestra.run.smithi118.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-16T07:49:42.822 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-16T07:49:42.877 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-16T07:49:42.877 DEBUG:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-16T07:49:42.932 INFO:tasks.workunit:timeout=3h 2024-04-16T07:49:42.933 INFO:tasks.workunit:cleanup=True 2024-04-16T07:49:42.933 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a7639cb6ce633dabc4d37769d62caacb54582984 2024-04-16T07:49:42.988 INFO:tasks.workunit.client.0.smithi118.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-16T07:50:25.768 INFO:tasks.workunit.client.0.smithi118.stderr:Note: switching to 'a7639cb6ce633dabc4d37769d62caacb54582984'. 2024-04-16T07:50:25.768 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-04-16T07:50:25.768 INFO:tasks.workunit.client.0.smithi118.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-16T07:50:25.768 INFO:tasks.workunit.client.0.smithi118.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-16T07:50:25.768 INFO:tasks.workunit.client.0.smithi118.stderr:state without impacting any branches by switching back to a branch. 2024-04-16T07:50:25.768 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-04-16T07:50:25.768 INFO:tasks.workunit.client.0.smithi118.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr: git switch -c 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr:Or undo this operation with: 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr: git switch - 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-04-16T07:50:25.769 INFO:tasks.workunit.client.0.smithi118.stderr:HEAD is now at a7639cb6ce6 Merge pull request #56885 from Svelar/asan_OSDScrub 2024-04-16T07:50:25.770 DEBUG:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-16T07:50:25.836 INFO:tasks.workunit.client.0.smithi118.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-16T07:50:25.838 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-16T07:50:25.839 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-16T07:50:26.512 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-16T07:50:26.575 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-16T07:50:26.616 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-16T07:50:26.618 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-16T07:50:26.618 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-16T07:50:26.657 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-16T07:50:26.660 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-04-16T07:50:26.660 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-16T07:50:26.716 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2024-04-16T07:50:26.717 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2024-04-16T07:50:26.718 DEBUG:teuthology.orchestra.run.smithi118:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7639cb6ce633dabc4d37769d62caacb54582984 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-04-16T07:50:26.838 INFO:tasks.workunit.client.0.smithi118.stdout:seed 34971 2024-04-16T07:50:26.838 INFO:tasks.workunit.client.0.smithi118.stdout:Note: Google Test filter = TestLibRBD.* 2024-04-16T07:50:26.838 INFO:tasks.workunit.client.0.smithi118.stdout:[==========] Running 168 tests from 1 test suite. 2024-04-16T07:50:26.838 INFO:tasks.workunit.client.0.smithi118.stdout:[----------] Global test environment set-up. 2024-04-16T07:50:26.838 INFO:tasks.workunit.client.0.smithi118.stdout:[----------] 168 tests from TestLibRBD 2024-04-16T07:50:26.884 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-04-16T07:50:28.593 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:30.779 INFO:tasks.workunit.client.0.smithi118.stdout:image has size 2097152 and order 22 2024-04-16T07:50:30.780 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.CreateAndStat (3896 ms) 2024-04-16T07:50:30.780 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-04-16T07:50:30.780 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:780: Skipped 2024-04-16T07:50:30.780 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:50:30.780 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool (0 ms) 2024-04-16T07:50:30.780 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-04-16T07:50:30.780 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:30.861 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.CreateAndStatPP (82 ms) 2024-04-16T07:50:30.861 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.GetId 2024-04-16T07:50:30.861 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:30.938 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.GetId (76 ms) 2024-04-16T07:50:30.938 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-04-16T07:50:30.938 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:31.059 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.GetIdPP (122 ms) 2024-04-16T07:50:31.059 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-04-16T07:50:31.060 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:31.135 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (75 ms) 2024-04-16T07:50:31.135 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-04-16T07:50:31.135 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (76 ms) 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:939: Skipped 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp (0 ms) 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:962: Skipped 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP (0 ms) 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.OpenAio 2024-04-16T07:50:31.211 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:31.290 INFO:tasks.workunit.client.0.smithi118.stdout:image has size 2097152 and order 22 2024-04-16T07:50:31.290 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.OpenAio (79 ms) 2024-04-16T07:50:31.290 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-04-16T07:50:31.301 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.OpenAioFail (12 ms) 2024-04-16T07:50:31.301 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-04-16T07:50:31.301 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:31.431 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.OpenAioPP (129 ms) 2024-04-16T07:50:31.431 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-04-16T07:50:31.442 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.OpenAioFailPP (11 ms) 2024-04-16T07:50:31.442 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-04-16T07:50:31.442 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:31.634 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.ResizeAndStat (193 ms) 2024-04-16T07:50:31.635 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-04-16T07:50:31.635 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:31.788 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (154 ms) 2024-04-16T07:50:31.789 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-04-16T07:50:31.789 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:31.973 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (184 ms) 2024-04-16T07:50:31.973 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-04-16T07:50:31.973 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:32.155 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (183 ms) 2024-04-16T07:50:32.156 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-04-16T07:50:33.861 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:35.826 INFO:tasks.workunit.client.0.smithi118.stdout:image: image15 2024-04-16T07:50:35.827 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image15 2024-04-16T07:50:35.827 INFO:tasks.workunit.client.0.smithi118.stdout:found image15 2024-04-16T07:50:35.827 INFO:tasks.workunit.client.0.smithi118.stdout:erased image15 2024-04-16T07:50:35.827 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:36.039 INFO:tasks.workunit.client.0.smithi118.stdout:image: image15 2024-04-16T07:50:36.039 INFO:tasks.workunit.client.0.smithi118.stdout:image: image16 2024-04-16T07:50:36.039 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image15 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:found image15 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:erased image15 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image16 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:found image16 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:erased image16 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:image: image16 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image16 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:found image16 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:erased image16 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (3883 ms) 2024-04-16T07:50:36.040 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-04-16T07:50:37.624 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:39.891 INFO:tasks.workunit.client.0.smithi118.stdout:num images is: 1 2024-04-16T07:50:39.891 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:50:39.891 INFO:tasks.workunit.client.0.smithi118.stdout:image: image17 2024-04-16T07:50:39.891 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image17 2024-04-16T07:50:39.891 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:39.934 INFO:tasks.workunit.client.0.smithi118.stdout:num images is: 2 2024-04-16T07:50:39.934 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 2 2024-04-16T07:50:39.934 INFO:tasks.workunit.client.0.smithi118.stdout:image: image17 2024-04-16T07:50:39.934 INFO:tasks.workunit.client.0.smithi118.stdout:image: image18 2024-04-16T07:50:39.934 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image17 2024-04-16T07:50:39.934 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image18 2024-04-16T07:50:40.035 INFO:tasks.workunit.client.0.smithi118.stdout:num images is: 1 2024-04-16T07:50:40.035 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:50:40.035 INFO:tasks.workunit.client.0.smithi118.stdout:image: image18 2024-04-16T07:50:40.035 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image18 2024-04-16T07:50:40.035 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (3995 ms) 2024-04-16T07:50:40.035 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCopy 2024-04-16T07:50:41.636 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:image: image19 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image19 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:found image19 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:erased image19 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:image: image19 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:image: image20 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image19 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:found image19 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:erased image19 2024-04-16T07:50:49.297 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image20 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:found image20 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:erased image20 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:0.00% done 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:100.00% done 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:image: image19 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:image: image20 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:image: image21 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image19 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:found image19 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:erased image19 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image20 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:found image20 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:erased image20 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image21 2024-04-16T07:50:49.298 INFO:tasks.workunit.client.0.smithi118.stdout:found image21 2024-04-16T07:50:49.299 INFO:tasks.workunit.client.0.smithi118.stdout:erased image21 2024-04-16T07:50:49.299 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCopy (9262 ms) 2024-04-16T07:50:49.299 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-04-16T07:50:50.664 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:50:55.090 INFO:tasks.workunit.client.0.smithi118.stdout:num images is: 1 2024-04-16T07:50:55.090 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:50:55.090 INFO:tasks.workunit.client.0.smithi118.stdout:image: image22 2024-04-16T07:50:55.090 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image22 2024-04-16T07:50:57.168 INFO:tasks.workunit.client.0.smithi118.stdout:num images is: 2 2024-04-16T07:50:57.169 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 2 2024-04-16T07:50:57.169 INFO:tasks.workunit.client.0.smithi118.stdout:image: image22 2024-04-16T07:50:57.169 INFO:tasks.workunit.client.0.smithi118.stdout:image: image23 2024-04-16T07:50:57.169 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image22 2024-04-16T07:50:57.169 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image23 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:0.00% done 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:100.00% done 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:num images is: 3 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 3 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:image: image22 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:image: image23 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:image: image24 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image22 2024-04-16T07:50:57.800 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image23 2024-04-16T07:50:57.801 INFO:tasks.workunit.client.0.smithi118.stdout:expected = image24 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCopyPP (9009 ms) 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:1578: Skipped 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopy (0 ms) 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:1745: Skipped 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:50:58.306 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.TestDeepCopyPP (0 ms) 2024-04-16T07:50:58.307 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-04-16T07:50:58.307 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:02.044 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 1 2024-04-16T07:51:02.044 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap1 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:found snap1 with size 2097152 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 2 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 2 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap1 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap2 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:found snap1 with size 2097152 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:found snap2 with size 4194304 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 1 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap2 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:found snap2 with size 4194304 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 0 2024-04-16T07:51:02.045 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 0 2024-04-16T07:51:02.046 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (3738 ms) 2024-04-16T07:51:02.046 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-04-16T07:51:02.046 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:1890: Skipped 2024-04-16T07:51:02.046 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:51:02.046 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.TestGetSnapShotTimeStamp (0 ms) 2024-04-16T07:51:02.046 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-04-16T07:51:02.046 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:02.894 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 1 2024-04-16T07:51:02.894 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:51:02.894 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap1 2024-04-16T07:51:02.895 INFO:tasks.workunit.client.0.smithi118.stdout:found snap1 with size 2097152 2024-04-16T07:51:03.842 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 2 2024-04-16T07:51:03.842 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 2 2024-04-16T07:51:03.842 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap1 2024-04-16T07:51:03.842 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap2 2024-04-16T07:51:03.842 INFO:tasks.workunit.client.0.smithi118.stdout:found snap1 with size 2097152 2024-04-16T07:51:03.842 INFO:tasks.workunit.client.0.smithi118.stdout:found snap2 with size 4194304 2024-04-16T07:51:04.861 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 1 2024-04-16T07:51:04.861 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:51:04.861 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap2 2024-04-16T07:51:04.861 INFO:tasks.workunit.client.0.smithi118.stdout:found snap2 with size 4194304 2024-04-16T07:51:06.026 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 0 2024-04-16T07:51:06.026 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 0 2024-04-16T07:51:06.044 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4000 ms) 2024-04-16T07:51:06.044 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-04-16T07:51:06.044 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:11.908 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 0 2024-04-16T07:51:11.908 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 0 2024-04-16T07:51:11.919 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (5875 ms) 2024-04-16T07:51:11.919 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-04-16T07:51:11.919 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:12.873 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 1 2024-04-16T07:51:12.873 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:51:12.874 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap1 2024-04-16T07:51:12.874 INFO:tasks.workunit.client.0.smithi118.stdout:found snap1 with size 2097152 2024-04-16T07:51:14.051 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 2 2024-04-16T07:51:14.052 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 2 2024-04-16T07:51:14.052 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap1 2024-04-16T07:51:14.052 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap2 2024-04-16T07:51:14.052 INFO:tasks.workunit.client.0.smithi118.stdout:found snap1 with size 2097152 2024-04-16T07:51:14.052 INFO:tasks.workunit.client.0.smithi118.stdout:found snap2 with size 4194304 2024-04-16T07:51:14.088 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 2 2024-04-16T07:51:14.088 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 2 2024-04-16T07:51:14.088 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap1-rename 2024-04-16T07:51:14.088 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap2 2024-04-16T07:51:14.088 INFO:tasks.workunit.client.0.smithi118.stdout:found snap1-rename with size 2097152 2024-04-16T07:51:14.088 INFO:tasks.workunit.client.0.smithi118.stdout:found snap2 with size 4194304 2024-04-16T07:51:14.914 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 1 2024-04-16T07:51:14.915 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 1 2024-04-16T07:51:14.915 INFO:tasks.workunit.client.0.smithi118.stdout:snap: snap2-rename 2024-04-16T07:51:14.915 INFO:tasks.workunit.client.0.smithi118.stdout:found snap2-rename with size 4194304 2024-04-16T07:51:15.882 INFO:tasks.workunit.client.0.smithi118.stdout:num snaps is: 0 2024-04-16T07:51:15.883 INFO:tasks.workunit.client.0.smithi118.stdout:expected: 0 2024-04-16T07:51:15.894 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (3975 ms) 2024-04-16T07:51:15.894 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-04-16T07:51:17.751 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:17.751 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:17.751 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:17.751 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:20.472 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (4578 ms) 2024-04-16T07:51:20.472 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-04-16T07:51:22.323 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:24.189 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:51:41.654 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.RemoveFullTry (21181 ms) 2024-04-16T07:51:41.654 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-04-16T07:51:41.655 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:2206: Skipped 2024-04-16T07:51:41.655 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:51:41.655 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.RemoveFullTryDataPool (0 ms) 2024-04-16T07:51:41.655 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestIO 2024-04-16T07:51:41.655 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:52:01.612 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.613 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.614 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.615 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:01.616 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:01.617 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.618 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:discard: 512~512 = 512 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:aio discard: 1536~512 = 0 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.619 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.620 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:52:01.621 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.622 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.623 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:01.624 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.625 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.626 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:01.627 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:01.627 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:01.627 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:01.627 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:01.627 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:01.627 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:01.627 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.866 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.866 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.866 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.866 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.867 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.868 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.869 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.870 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.871 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.872 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.873 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:18.874 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:18.875 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:18.875 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:18.875 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:18.875 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:18.875 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestIO (37213 ms) 2024-04-16T07:52:18.875 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-04-16T07:52:18.875 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:52:40.742 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:40.742 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:40.742 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:40.742 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:40.742 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:52:40.743 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.743 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:40.743 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.743 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.743 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:40.743 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.743 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:40.744 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.744 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.744 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:40.744 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.744 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:40.744 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.744 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.745 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:40.745 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.745 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:40.745 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.745 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.745 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:40.745 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.746 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:52:40.746 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.746 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.746 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:52:40.746 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:40.746 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:40.746 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:40.747 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:40.747 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:40.747 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:40.747 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:40.747 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:40.747 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:52:40.747 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:52:40.748 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.748 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:40.748 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.748 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.748 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:40.748 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.748 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:40.749 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.749 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.749 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:40.749 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.749 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:40.749 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.749 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.750 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:40.750 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.750 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:40.750 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.750 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.750 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:40.751 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.751 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:52:40.751 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.751 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.751 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:52:40.751 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.752 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.752 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.752 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.752 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.752 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.752 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:40.753 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:40.753 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:40.753 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.753 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:40.753 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:40.753 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:40.753 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.754 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:40.754 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:40.754 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:40.754 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.754 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:40.754 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:40.754 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:40.754 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:discard: 512~512 = 512 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:aio discard: 1536~512 = 0 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.755 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.756 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.756 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:52:40.756 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:52:40.756 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.756 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.756 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:52:40.756 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.757 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.757 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:52:40.757 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.757 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.757 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:52:40.757 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.758 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.758 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.758 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.758 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:52:40.758 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.758 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.758 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:52:40.759 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.759 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.759 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:52:40.759 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.759 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.759 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:52:40.760 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.760 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.760 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.760 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.760 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:52:40.760 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.761 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.761 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:52:40.761 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.761 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.761 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:52:40.761 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.762 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.762 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:52:40.762 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.762 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.762 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.762 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:52:40.763 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.764 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.765 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.766 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.767 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:52:40.768 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.877 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.878 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.879 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.880 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.881 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.882 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:00.883 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.884 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:00.885 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 4 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:read: 4 2024-04-16T07:53:00.886 INFO:tasks.workunit.client.0.smithi118.stdout:read: 4 2024-04-16T07:53:00.887 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (42011 ms) 2024-04-16T07:53:00.887 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-04-16T07:53:00.887 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:53:29.045 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:53:29.045 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:53:29.045 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:53:29.045 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:53:29.045 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:53:29.045 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.045 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.046 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:53:29.047 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:53:29.048 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:53:29.049 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:53:29.050 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:53:29.051 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:discard: 512~512 = 512 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:aio discard: 1536~512 = 0 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.052 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:53:29.053 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.054 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.055 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.056 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.057 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.058 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.059 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:29.060 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:29.061 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:29.061 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.670 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.670 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.670 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.670 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.670 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.670 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.670 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.670 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.671 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.672 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:50.673 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.674 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.675 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.676 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.677 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:50.678 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 4 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:read: 4 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:read: 4 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (49792 ms) 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:2406: Skipped 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.TestCloneEncryption (0 ms) 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:2594: Skipped 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (0 ms) 2024-04-16T07:53:50.679 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-04-16T07:53:50.680 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:2670: Skipped 2024-04-16T07:53:50.680 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:53:50.680 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (0 ms) 2024-04-16T07:53:50.680 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-04-16T07:53:50.680 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:53:54.985 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4316 ms) 2024-04-16T07:53:54.985 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-04-16T07:53:54.986 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:00.831 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (5845 ms) 2024-04-16T07:54:00.831 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-04-16T07:54:00.831 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:2840: Skipped 2024-04-16T07:54:00.831 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:54:00.831 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2024-04-16T07:54:00.831 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-04-16T07:54:00.832 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:2920: Skipped 2024-04-16T07:54:00.832 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:54:00.832 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadFormatMismatch (0 ms) 2024-04-16T07:54:00.832 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-04-16T07:54:00.832 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.EncryptedResize (14874 ms) 2024-04-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-04-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:3137: Skipped 2024-04-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.EncryptedFlattenSmallData (0 ms) 2024-04-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-04-16T07:54:15.705 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:32.996 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:32.997 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:32.998 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:32.999 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:33.000 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:33.001 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:33.002 INFO:tasks.workunit.client.0.smithi118.stdout:discard: 512~512 = 512 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:aio discard: 1536~512 = 0 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:54:33.003 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.004 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.005 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.006 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.007 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.008 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.009 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:33.010 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:33.011 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:33.011 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:33.011 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:33.011 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:33.011 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:33.011 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.895 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.895 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:47.896 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.897 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.898 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.899 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.900 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.901 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.902 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:47.903 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (32190 ms) 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3109-g30b0aa00/rpm/el9/BUILD/ceph-19.0.0-3109-g30b0aa00/src/test/librbd/test_librbd.cc:3882: Skipped 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:Skipping due to unmet REQUIRE 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:[ SKIPPED ] TestLibRBD.TestDataPoolIO (0 ms) 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-04-16T07:54:47.904 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:48.005 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (111 ms) 2024-04-16T07:54:48.005 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-04-16T07:54:48.005 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:48.111 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (106 ms) 2024-04-16T07:54:48.111 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-04-16T07:54:48.111 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:48.231 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (120 ms) 2024-04-16T07:54:48.232 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-04-16T07:54:48.232 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:48.345 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (114 ms) 2024-04-16T07:54:48.346 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-04-16T07:54:48.346 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:48.498 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (153 ms) 2024-04-16T07:54:48.498 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-04-16T07:54:48.498 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:48.649 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (151 ms) 2024-04-16T07:54:48.649 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-04-16T07:54:48.650 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:48.824 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (175 ms) 2024-04-16T07:54:48.825 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-04-16T07:54:48.825 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:48.966 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (142 ms) 2024-04-16T07:54:48.966 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-04-16T07:54:48.966 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:51.267 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (2301 ms) 2024-04-16T07:54:51.267 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-04-16T07:54:51.267 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (2190 ms) 2024-04-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-04-16T07:54:53.458 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:53.570 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (113 ms) 2024-04-16T07:54:53.571 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-04-16T07:54:53.571 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:53.686 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (116 ms) 2024-04-16T07:54:53.686 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-04-16T07:54:53.686 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:53.796 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (110 ms) 2024-04-16T07:54:53.796 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-04-16T07:54:53.796 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:53.898 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (102 ms) 2024-04-16T07:54:53.898 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-04-16T07:54:53.898 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:54.006 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.006 INFO:tasks.workunit.client.0.smithi118.stdout:aio discard: 0~1048576 = 0 2024-04-16T07:54:54.007 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.007 INFO:tasks.workunit.client.0.smithi118.stdout:aio discard: 0~4194304 = 0 2024-04-16T07:54:54.007 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.007 INFO:tasks.workunit.client.0.smithi118.stdout:aio discard: 3145728~1048576 = 0 2024-04-16T07:54:54.007 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (108 ms) 2024-04-16T07:54:54.007 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestFUA 2024-04-16T07:54:54.007 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:54.436 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.436 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.436 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.436 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.436 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.436 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.437 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.438 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.439 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestFUA (430 ms) 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-04-16T07:54:54.440 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:54:54.501 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.501 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.502 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:54:54.503 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.662 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.662 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:54:54.662 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:start compare and write 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:compare and wrote: 512 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:54.663 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.673 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.673 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:54.673 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.674 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:54.674 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.684 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.684 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:54.684 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.684 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:54.685 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.696 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.696 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:54.696 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.696 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:54.696 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.707 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.707 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:54.707 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.707 INFO:tasks.workunit.client.0.smithi118.stdout:started aio compare and write 2024-04-16T07:54:54.707 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:54:54.732 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:finished aio compare and write 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.733 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.736 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.736 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:54:54.736 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.736 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.736 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.740 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.740 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:54:54.740 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.740 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.740 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.744 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.744 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:54:54.744 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.744 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.744 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:discard: 512~512 2024-04-16T07:54:54.748 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:11.033 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.034 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:55:11.035 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:55:11.036 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.037 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.038 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.039 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:11.040 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:11.040 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:11.040 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.040 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.040 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.040 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.040 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:11.162 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:11.162 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:11.163 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.163 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.163 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.163 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.163 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:11.281 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:11.281 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:11.281 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.281 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.282 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:11.648 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:11.648 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:11.648 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.648 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.649 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.649 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.649 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:11.988 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:11.988 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:11.988 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:11.988 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:11.988 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:11.988 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:11.988 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:12.479 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:12.479 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:12.479 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:12.479 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:12.479 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:12.479 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:12.479 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:12.935 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:12.935 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:12.935 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:12.935 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:12.935 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:12.936 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:12.936 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:12.936 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:12.936 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:12.936 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:12.936 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:12.936 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:12.936 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:13.664 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:13.664 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:13.665 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:13.665 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:13.665 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:13.665 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:13.665 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:14.516 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:14.516 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:14.516 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:14.516 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:14.516 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:14.516 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:14.516 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:15.443 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:15.444 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:15.444 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:15.444 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:15.444 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:15.444 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:15.444 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:16.208 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:16.208 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:16.208 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:16.209 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:17.365 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:17.365 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:17.365 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:17.365 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:17.365 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:17.366 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:17.366 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:18.508 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:18.508 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:18.508 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:18.508 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:18.508 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:18.508 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:18.508 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:19.711 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:19.711 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:19.711 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:19.711 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:19.712 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:19.712 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:19.712 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:20.799 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:20.800 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:22.448 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:22.448 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:22.448 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:22.448 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:22.448 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:22.448 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:22.448 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:23.847 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:23.847 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:23.847 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:23.847 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:23.847 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:23.847 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:23.848 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:25.487 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:25.487 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:25.487 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:25.487 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:25.487 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:25.487 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:25.487 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:27.065 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:27.065 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:27.065 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestIOPP (32629 ms) 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-04-16T07:55:27.066 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:27.166 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (101 ms) 2024-04-16T07:55:27.166 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-04-16T07:55:27.166 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:27.261 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (95 ms) 2024-04-16T07:55:27.261 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-04-16T07:55:27.261 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:27.355 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (94 ms) 2024-04-16T07:55:27.355 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-04-16T07:55:27.355 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:27.451 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (96 ms) 2024-04-16T07:55:27.452 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-04-16T07:55:27.452 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:27.565 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (114 ms) 2024-04-16T07:55:27.565 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-04-16T07:55:27.565 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:27.653 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:27.676 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (111 ms) 2024-04-16T07:55:27.676 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-04-16T07:55:27.677 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:27.783 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (107 ms) 2024-04-16T07:55:27.783 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-04-16T07:55:27.783 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:27.875 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:27.900 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (117 ms) 2024-04-16T07:55:27.900 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-04-16T07:55:27.900 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:28.011 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (111 ms) 2024-04-16T07:55:28.011 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-04-16T07:55:28.011 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:28.101 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:28.124 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (113 ms) 2024-04-16T07:55:28.125 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-04-16T07:55:28.125 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:28.248 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (124 ms) 2024-04-16T07:55:28.248 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-04-16T07:55:28.248 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:28.353 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:28.387 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (139 ms) 2024-04-16T07:55:28.387 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-04-16T07:55:28.387 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:28.545 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (157 ms) 2024-04-16T07:55:28.545 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-04-16T07:55:28.545 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:28.658 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:28.690 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (145 ms) 2024-04-16T07:55:28.690 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-04-16T07:55:28.690 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:28.841 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (152 ms) 2024-04-16T07:55:28.842 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-04-16T07:55:28.842 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:28.965 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:29.000 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (159 ms) 2024-04-16T07:55:29.000 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-04-16T07:55:29.001 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:31.284 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (2283 ms) 2024-04-16T07:55:31.284 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-04-16T07:55:31.284 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:33.424 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:33.466 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (2183 ms) 2024-04-16T07:55:33.466 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-04-16T07:55:33.466 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:55:33.600 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:55:33.600 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:55:33.601 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:55:33.601 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:55:33.601 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 512 2024-04-16T07:55:33.601 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.601 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:55:33.601 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:33.610 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:33.610 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:55:33.610 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.610 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:55:33.610 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:33.621 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:33.621 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:55:33.621 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.621 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:55:33.621 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:33.631 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:33.631 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:55:33.631 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.631 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:55:33.632 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:33.641 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:33.642 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:55:33.642 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.642 INFO:tasks.workunit.client.0.smithi118.stdout:started write 2024-04-16T07:55:33.642 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:33.653 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:33.653 INFO:tasks.workunit.client.0.smithi118.stdout:finished write 2024-04-16T07:55:33.653 INFO:tasks.workunit.client.0.smithi118.stdout:read: 512 2024-04-16T07:55:33.653 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.653 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:55:33.653 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:55:33.658 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:55:33.658 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:55:33.658 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.658 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:55:33.658 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:55:33.662 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:55:33.662 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:55:33.662 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.662 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:55:33.662 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:55:33.666 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:55:33.666 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:55:33.666 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.666 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:55:33.666 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:55:33.670 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:55:33.670 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:55:33.670 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:33.670 INFO:tasks.workunit.client.0.smithi118.stdout:started read 2024-04-16T07:55:33.670 INFO:tasks.workunit.client.0.smithi118.stdout:read completion cb called! 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 512 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:finished read 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 0 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.731 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 16384 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 49152 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.732 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 65536 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 98304 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 114688 2024-04-16T07:55:49.733 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 147456 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 163840 2024-04-16T07:55:49.734 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 196608 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:wrote: 212992 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.735 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:49.736 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:49.741 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:49.741 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:49.741 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.741 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.741 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:49.741 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:49.741 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:49.869 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:49.869 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:49.869 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:49.869 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:49.869 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:49.870 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:49.870 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:50.054 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:50.054 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:50.054 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:50.054 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:50.054 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:50.054 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:50.054 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:50.054 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:50.055 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:50.055 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:50.055 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:50.055 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:50.055 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:50.540 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:50.541 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:50.541 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:50.541 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:50.541 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:50.541 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:50.541 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:50.874 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:50.874 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:50.874 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:50.874 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:50.874 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:50.874 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:50.874 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:51.360 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:51.361 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:51.361 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:51.361 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:51.361 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:51.361 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:51.361 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:51.804 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:51.805 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:51.805 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:52.532 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:52.533 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:52.533 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:52.533 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:52.533 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:52.533 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:52.533 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:53.189 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:53.189 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:53.189 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:53.189 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:53.190 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:53.190 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:53.190 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:54.114 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:54.115 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:54.115 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:54.115 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:54.115 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:54.115 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:54.115 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:55.003 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:55.004 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:56.163 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:56.163 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:56.163 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:56.163 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:56.163 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:56.164 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:56.164 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:57.216 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:57.216 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:57.216 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:57.216 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:57.216 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:57.216 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:57.216 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:58.558 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:58.559 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:55:59.658 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:55:59.659 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:56:01.108 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:56:01.108 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:56:01.108 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:56:01.108 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:56:01.108 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:56:01.108 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:56:01.108 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:56:02.622 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:56:02.622 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:56:02.622 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:56:02.622 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:56:02.622 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:56:02.622 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:56:02.622 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:56:04.262 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:56:04.262 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:56:04.262 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:56:04.262 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:56:04.262 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:56:04.262 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:56:04.262 INFO:tasks.workunit.client.0.smithi118.stdout:write completion cb called! 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:return value is: 0 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:finished writesame 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:to verify the data 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:verified 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:created completion 2024-04-16T07:56:05.764 INFO:tasks.workunit.client.0.smithi118.stdout:started writesame 2024-04-16T07:56:05.765 INFO:tasks.workunit.client.0.smithi118.stdout:expected fail, finished writesame 2024-04-16T07:56:05.765 INFO:tasks.workunit.client.0.smithi118.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (32298 ms) 2024-04-16T07:56:05.765 INFO:tasks.workunit.client.0.smithi118.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-04-16T07:56:05.765 INFO:tasks.workunit.client.0.smithi118.stdout:using old format 2024-04-16T07:56:14.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-16T07:56:19.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-04-16T07:56:24.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2024-04-16T07:56:30.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2024-04-16T07:56:35.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2024-04-16T07:56:41.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2024-04-16T07:56:46.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2024-04-16T07:56:46.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-16T07:56:51.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~38s 2024-04-16T07:56:51.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-04-16T07:56:56.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~43s 2024-04-16T07:56:56.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-04-16T07:57:02.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2024-04-16T07:57:02.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-04-16T07:57:07.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~54s 2024-04-16T07:57:07.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~21s 2024-04-16T07:57:12.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~59s 2024-04-16T07:57:12.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2024-04-16T07:57:18.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2024-04-16T07:57:18.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2024-04-16T07:57:23.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~70s 2024-04-16T07:57:23.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~37s 2024-04-16T07:57:28.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~75s 2024-04-16T07:57:28.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~42s 2024-04-16T07:57:34.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2024-04-16T07:57:34.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~48s 2024-04-16T07:57:39.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2024-04-16T07:57:39.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~53s 2024-04-16T07:57:40.630 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:57:40.630+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:57:44.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~91s 2024-04-16T07:57:44.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~58s 2024-04-16T07:57:45.630 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:57:45.631+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:57:49.995 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~96s 2024-04-16T07:57:49.995 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~64s 2024-04-16T07:57:50.631 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:57:50.631+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:57:55.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2024-04-16T07:57:55.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~69s 2024-04-16T07:57:55.632 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:57:55.632+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:00.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~107s 2024-04-16T07:58:00.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~74s 2024-04-16T07:58:00.633 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:00.633+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:05.634 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:05.634+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:05.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~112s 2024-04-16T07:58:05.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~80s 2024-04-16T07:58:10.635 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:10.634+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:11.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2024-04-16T07:58:11.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~85s 2024-04-16T07:58:15.635 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:15.635+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:16.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~123s 2024-04-16T07:58:16.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2024-04-16T07:58:20.636 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:20.636+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:21.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~128s 2024-04-16T07:58:21.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~95s 2024-04-16T07:58:25.637 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:25.637+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:27.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2024-04-16T07:58:27.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~101s 2024-04-16T07:58:30.638 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:30.639+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:32.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2024-04-16T07:58:32.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~106s 2024-04-16T07:58:35.639 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:35.639+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:37.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~144s 2024-04-16T07:58:37.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~111s 2024-04-16T07:58:40.640 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:40.639+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:43.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~149s 2024-04-16T07:58:43.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~117s 2024-04-16T07:58:45.641 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:45.641+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:48.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2024-04-16T07:58:48.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~122s 2024-04-16T07:58:50.642 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:50.641+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:53.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2024-04-16T07:58:53.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~127s 2024-04-16T07:58:55.642 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:58:55.642+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:58:58.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2024-04-16T07:58:58.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~133s 2024-04-16T07:59:00.643 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:00.643+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:04.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2024-04-16T07:59:04.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~138s 2024-04-16T07:59:05.644 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:05.644+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:09.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~176s 2024-04-16T07:59:09.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~143s 2024-04-16T07:59:10.645 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:10.645+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:14.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~181s 2024-04-16T07:59:14.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~148s 2024-04-16T07:59:15.646 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:15.647+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:20.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2024-04-16T07:59:20.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~154s 2024-04-16T07:59:20.647 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:20.646+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:25.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~191s 2024-04-16T07:59:25.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~159s 2024-04-16T07:59:25.648 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:25.648+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:30.648 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:30.648+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:30.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~197s 2024-04-16T07:59:30.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~164s 2024-04-16T07:59:35.649 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:35.649+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:36.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~202s 2024-04-16T07:59:36.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~170s 2024-04-16T07:59:40.650 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:40.650+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:41.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2024-04-16T07:59:41.352 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~175s 2024-04-16T07:59:45.651 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:45.651+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:46.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~213s 2024-04-16T07:59:46.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~180s 2024-04-16T07:59:50.652 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:50.652+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:51.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~218s 2024-04-16T07:59:51.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~186s 2024-04-16T07:59:55.652 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T07:59:55.652+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T07:59:57.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~223s 2024-04-16T07:59:57.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~191s 2024-04-16T08:00:00.653 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:00.654+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:02.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~229s 2024-04-16T08:00:02.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~196s 2024-04-16T08:00:05.654 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:05.654+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:07.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~234s 2024-04-16T08:00:07.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~202s 2024-04-16T08:00:10.655 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:10.655+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:13.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~239s 2024-04-16T08:00:13.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~207s 2024-04-16T08:00:15.656 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:15.656+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:18.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~245s 2024-04-16T08:00:18.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~212s 2024-04-16T08:00:20.657 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:20.656+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:23.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~250s 2024-04-16T08:00:23.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~217s 2024-04-16T08:00:25.658 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:25.657+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:29.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2024-04-16T08:00:29.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~223s 2024-04-16T08:00:30.658 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:30.658+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:34.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2024-04-16T08:00:34.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~228s 2024-04-16T08:00:35.659 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:35.659+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:39.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~266s 2024-04-16T08:00:39.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~233s 2024-04-16T08:00:40.660 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:40.660+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:44.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~271s 2024-04-16T08:00:44.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~239s 2024-04-16T08:00:45.661 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:45.662+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:50.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2024-04-16T08:00:50.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~244s 2024-04-16T08:00:50.662 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:50.662+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:00:55.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~282s 2024-04-16T08:00:55.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~249s 2024-04-16T08:00:55.663 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:00:55.663+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:01:00.663 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:01:00.663+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:01:00.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~287s 2024-04-16T08:01:00.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~255s 2024-04-16T08:01:05.664 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:01:05.664+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:01:06.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~292s 2024-04-16T08:01:06.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~260s 2024-04-16T08:01:10.665 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:01:10.665+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:01:11.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~298s 2024-04-16T08:01:11.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2024-04-16T08:01:15.666 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:01:15.666+0000 7f56a5bf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.118:6806/2962547363 for 29sec e76 v76) 2024-04-16T08:01:16.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~303s 2024-04-16T08:01:16.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2024-04-16T08:01:16.802 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-16T08:01:17.104 INFO:tasks.ceph.osd.2:Sent signal 15 2024-04-16T08:01:17.105 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-16T08:01:17.105 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-16T08:01:17.106 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:01:17.106+0000 7f56a8bff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 32123) UID: 0 2024-04-16T08:01:17.106 INFO:tasks.ceph.mon.a.smithi118.stderr:2024-04-16T08:01:17.106+0000 7f56a8bff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T08:01:17.107 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T08:01:17.106+0000 7fefab1f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 32125) UID: 0 2024-04-16T08:01:17.107 INFO:tasks.ceph.mgr.x.smithi118.stderr:2024-04-16T08:01:17.106+0000 7fefab1f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-16T10:50:26.776 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T10:50:26.794 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2024-04-16T10:50:26.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-16T10:50:27.092 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi118 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7639cb6ce633dabc4d37769d62caacb54582984 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-04-16T10:50:27.298 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3e13731b6c054ce7a6440d1571391c56 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi118 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7639cb6ce633dabc4d37769d62caacb54582984 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-04-16T10:50:27.300 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T10:50:27.309 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T10:50:27.309 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T10:50:27.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-04-16T10:50:27.424+0000 7f7a5ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T10:52:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T10:52:27.348 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T10:52:27.359 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T10:52:27.359 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T10:52:27.360 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T10:52:27.360 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T10:52:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T10:52:27.362 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2024-04-16T10:52:27.371 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T10:52:27.372 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T10:52:27.372 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T10:52:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T10:52:27.372 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-04-16T10:52:27.381 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T10:52:27.381 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T10:52:27.381 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T10:52:27.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T10:52:27.381 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2024-04-16T10:52:27.392 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T10:52:27.392 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T10:52:27.392 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T10:52:27.392 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T10:52:27.392 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T10:52:27.393 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T10:52:27.393 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T10:52:27.393 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T10:52:27.393 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T10:52:27.393 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T10:52:27.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(POOL_FULL\)' | head -n 1 2024-04-16T10:52:27.489 INFO:teuthology.orchestra.run.smithi118.stdout:2024-04-16T07:56:39.747500+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T10:52:27.490 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-16T10:52:27.490 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T10:52:27.490 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T10:52:27.588 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T10:52:27.588 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T10:52:27.644 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T10:52:27.644 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T10:52:27.713 INFO:tasks.ceph:Archiving mon data... 2024-04-16T10:52:27.715 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987/data/mon.a.tgz 2024-04-16T10:52:27.715 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-04-16T10:52:27.731 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.4Zkwj0SRcf 2024-04-16T10:52:27.732 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.4Zkwj0SRcf 2024-04-16T10:52:27.879 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.4Zkwj0SRcf 2024-04-16T10:52:27.967 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.4Zkwj0SRcf is 233KB 2024-04-16T10:52:28.062 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.4Zkwj0SRcf 2024-04-16T10:52:28.076 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T10:52:28.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T10:52:28.200 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T10:52:28.200 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T10:52:28.202 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987/remote/smithi118/crash 2024-04-16T10:52:28.203 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T10:52:28.267 INFO:tasks.ceph:Compressing logs... 2024-04-16T10:52:28.267 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T10:52:28.345 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32365.log 2024-04-16T10:52:28.345 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5/var/log/ceph/ceph-client.admin.32365.log: --verbose -- /var/log/ceph/ceph-client.admin.34456.log 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32365.log.gz 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32405.log 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33996.log 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34456.log.gz 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.32405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34582.log 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32405.log.gz 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.33996.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T10:52:28.346 INFO:teuthology.orchestra.run.smithi118.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33996.log.gz 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34203.log 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34582.log.gz 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32215.log 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34203.log: gzip -5 --verbose -- 86.7% /var/log/ceph/ceph-osd.1.log -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34203.log.gz 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.32215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34329.log 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32215.log.gz 2024-04-16T10:52:28.347 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34752.log 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34416.log 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34329.log.gz 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32593.log 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34752.log.gz 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34710.log 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34416.log.gz 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.32593.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32593.log.gz 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34038.log 2024-04-16T10:52:28.348 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34290.log 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34710.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34710.log.gz 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34202.log 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34038.log.gz 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37866.log 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34290.log.gz 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34080.log 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34202.log.gz 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.37866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32563.log 2024-04-16T10:52:28.349 INFO:teuthology.orchestra.run.smithi118.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37866.log.gz 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34080.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28884.log 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34080.log.gz 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.32563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32325.log 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32563.log.gz 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.tmp-client.admin.28884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34455.log 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28884.log.gz 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.32325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34122.log 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32325.log.gz 2024-04-16T10:52:28.350 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34455.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34455.log.gz 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34971.log 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34122.log.gz 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34330.log 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34971.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr: 91.8%/var/log/ceph/ceph-client.admin.34330.log: -- replaced with /var/log/ceph/ceph-client.admin.34971.log.gz 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33935.log 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34330.log.gz 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T10:52:28.351 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.33935.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T10:52:28.352 INFO:teuthology.orchestra.run.smithi118.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33935.log.gz 2024-04-16T10:52:28.352 INFO:teuthology.orchestra.run.smithi118.stderr:gzip/var/log/ceph/ceph.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34668.log 2024-04-16T10:52:28.352 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34581.log 2024-04-16T10:52:28.352 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34668.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34668.log.gz 2024-04-16T10:52:28.352 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33879.log 2024-04-16T10:52:28.352 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34581.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34581.log.gz 2024-04-16T10:52:28.354 INFO:teuthology.orchestra.run.smithi118.stderr: 89.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-16T10:52:28.354 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32127.log 2024-04-16T10:52:28.354 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.33879.log: gzip -5 --verbose -- 49.2% -- replaced with /var/log/ceph/ceph-client.admin.33879.log.gz 2024-04-16T10:52:28.354 INFO:teuthology.orchestra.run.smithi118.stderr: /var/log/ceph/ceph-client.admin.34204.log 2024-04-16T10:52:28.355 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.32127.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32127.log.gz 2024-04-16T10:52:28.355 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34580.log 2024-04-16T10:52:28.355 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34204.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34204.log.gz 2024-04-16T10:52:28.355 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-client.admin.34580.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34580.log.gz 2024-04-16T10:52:28.439 INFO:teuthology.orchestra.run.smithi118.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T10:52:28.568 INFO:teuthology.orchestra.run.smithi118.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T10:52:30.223 INFO:teuthology.orchestra.run.smithi118.stderr: 96.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T10:52:30.299 INFO:teuthology.orchestra.run.smithi118.stderr: 96.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T10:52:31.247 INFO:teuthology.orchestra.run.smithi118.stderr: 97.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T10:52:31.249 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-04-16T10:52:31.249 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m2.927s 2024-04-16T10:52:31.249 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m6.780s 2024-04-16T10:52:31.249 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m0.337s 2024-04-16T10:52:31.249 INFO:tasks.ceph:Archiving logs... 2024-04-16T10:52:31.250 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987/remote/smithi118/log 2024-04-16T10:52:31.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T10:52:32.032 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T10:52:32.041 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T10:52:32.050 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T10:52:32.054 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T10:52:32.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T10:52:32.110 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T10:52:32.118 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T10:52:32.118 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T10:52:32.140 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-04-16T10:52:32.145 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T10:52:32.145 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-04-16T10:52:32.145 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv01.front.sepia.ceph.com 3 7 377 76 +418us[ +422us] +/- 60ms 2024-04-16T10:52:32.145 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 17 -1317us[-1317us] +/- 74ms 2024-04-16T10:52:32.145 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +746us[ +746us] +/- 70ms 2024-04-16T10:52:32.145 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T10:52:32.145 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T10:52:32.154 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T10:52:32.155 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T10:52:32.162 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T10:52:32.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T10:52:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T10:52:32.282 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T10:52:32.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T10:52:32.299 INFO:teuthology.task.internal:Duration was 11735.470736 seconds 2024-04-16T10:52:32.299 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T10:52:32.313 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T10:52:32.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T10:52:32.368 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T10:52:32.684 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T10:52:32.684 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-04-16T10:52:32.684 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T10:52:32.704 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T10:52:32.704 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T10:52:32.774 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T10:52:32.775 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T10:52:32.918 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T10:52:32.928 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T10:52:32.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T10:52:32.956 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T10:52:32.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T10:52:33.023 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-04-16T10:52:33.056 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T10:52:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T10:52:33.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T10:52:33.121 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T10:52:33.121 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987/remote/smithi118 2024-04-16T10:52:33.122 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T10:52:33.188 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T10:52:33.188 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T10:52:33.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T10:52:33.248 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T10:52:33.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T10:52:33.257 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T10:52:33.257 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T10:52:33.293 INFO:teuthology.orchestra.run.smithi118.stdout: 266042 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 10:52 /home/ubuntu/cephtest 2024-04-16T10:52:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: 401568 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 07:49 /home/ubuntu/cephtest/mnt.0 2024-04-16T10:52:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: 401569 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 07:50 /home/ubuntu/cephtest/mnt.0/client.0 2024-04-16T10:52:33.294 INFO:teuthology.orchestra.run.smithi118.stdout: 415948 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 07:50 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-16T10:52:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T10:52:33.295 INFO:teuthology.orchestra.run.smithi118.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T10:52:33.295 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_a7639cb6ce633dabc4d37769d62caacb54582984/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T10:52:33.306 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T10:52:33.316 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T10:52:33.326 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T10:52:33.326 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T10:52:33.351 DEBUG:teuthology.nuke:shortname: smithi118 2024-04-16T10:52:33.351 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T10:52:33.371 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-16_07:26:08-crimson-rados-wip-matanb-crimson-replicated-april-15-distro-crimson-smithi/7657987', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 07:27:17.555075', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNb2mDReFOOnitsN7Mx+s7up9mh1GZXUj+FWZM1MBiYPxPvo1m4sNSkILYTUclrN6TJ6htpyremOhQ7ClHxDqfI='} 2024-04-16T10:52:33.420 INFO:teuthology.orchestra.console.smithi118:Power off 2024-04-16T10:52:33.421 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T10:52:33.444 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2024-04-16T10:52:33.445 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T10:52:33.467 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-04-16T10:52:37.469 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T10:52:37.581 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-04-16T10:52:41.582 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T10:52:41.694 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2024-04-16T10:52:41.795 INFO:teuthology.orchestra.console.smithi118:Power off completed 2024-04-16T10:52:41.900 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} duration: 11735.470736026764 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi118 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7639cb6ce633dabc4d37769d62caacb54582984 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3e13731b6c054ce7a6440d1571391c56 status: fail success: false 2024-04-16T10:52:41.901 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T10:52:41.973 INFO:teuthology.run:FAIL