2023-10-18T03:31:14.611 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-18T03:31:14.612 DEBUG:teuthology.run:Teuthology command: teuthology --name sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi --archive /home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264 --description crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} --verbose --owner scheduled_sjust@teuthology -- /home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264/orig.config.yaml 2023-10-18T03:31:14.641 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T03:31:14.717 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264 branch: wip-crimson-scrub-testing-2023-10-17 description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} email: null first_in_suite: false job_id: '7431264' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-crimson-scrub-testing-2023-10-17 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson seastar smp: 3 debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(POOL_FULL\) sha1: fbf795c854382dc4ab29c95372096a3ba925ed95 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: fbf795c854382dc4ab29c95372096a3ba925ed95 workunit: branch: wip-crimson-scrub-testing-2023-10-17 env: CRIMSON_COMPAT: '1' sha1: fbf795c854382dc4ab29c95372096a3ba925ed95 owner: scheduled_sjust@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4400 sha1: fbf795c854382dc4ab29c95372096a3ba925ed95 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-crimson-scrub-testing-2023-10-17 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: fbf795c854382dc4ab29c95372096a3ba925ed95 targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5XHdjHt2RFShvGSuGm1VMFOoHPjtSMZgUHDtXvvx4lESLwrSniIqeicsTbOvTtgUbPojTArnahW1WOutSZ4tw= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh env: RBD_FEATURES: '61' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-18_03:24:35 tube: smithi user: sjust verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T03:31:14.718 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa; will attempt to use it 2023-10-18T03:31:14.719 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks 2023-10-18T03:31:14.719 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T03:31:14.721 INFO:teuthology.task.internal:Checking packages... 2023-10-18T03:31:14.747 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'fbf795c854382dc4ab29c95372096a3ba925ed95' 2023-10-18T03:31:14.747 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T03:31:14.747 INFO:teuthology.packaging:ref: None 2023-10-18T03:31:14.748 INFO:teuthology.packaging:tag: None 2023-10-18T03:31:14.748 INFO:teuthology.packaging:branch: wip-crimson-scrub-testing-2023-10-17 2023-10-18T03:31:14.748 INFO:teuthology.packaging:sha1: fbf795c854382dc4ab29c95372096a3ba925ed95 2023-10-18T03:31:14.749 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-crimson-scrub-testing-2023-10-17 2023-10-18T03:31:14.893 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6794.gfbf795c8 2023-10-18T03:31:14.894 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T03:31:14.901 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T03:31:14.901 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T03:31:14.910 INFO:teuthology.task.internal:Saving configuration 2023-10-18T03:31:14.921 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T03:31:14.928 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T03:31:14.953 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 03:26:29.627168', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5XHdjHt2RFShvGSuGm1VMFOoHPjtSMZgUHDtXvvx4lESLwrSniIqeicsTbOvTtgUbPojTArnahW1WOutSZ4tw='} 2023-10-18T03:31:14.953 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T03:31:14.961 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-18T03:31:14.961 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T03:31:15.005 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faeefc75940>, signals=[15]) 2023-10-18T03:31:15.006 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T03:31:15.012 INFO:teuthology.task.internal:Opening connections... 2023-10-18T03:31:15.013 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T03:31:15.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T03:31:15.115 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T03:31:15.122 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2023-10-18T03:31:15.187 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2023-10-18T03:31:15.188 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2023-10-18T03:31:15.246 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2023-10-18T03:31:15.246 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2023-10-18T03:31:15.247 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2023-10-18T03:31:15.247 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2023-10-18T03:31:15.247 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2023-10-18T03:31:15.248 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2023-10-18T03:31:15.248 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-18T03:31:15.248 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2023-10-18T03:31:15.249 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2023-10-18T03:31:15.249 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-18T03:31:15.250 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2023-10-18T03:31:15.250 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T03:31:15.250 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-18T03:31:15.251 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T03:31:15.252 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-10-18T03:31:15.273 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T03:31:15.286 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T03:31:15.294 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T03:31:15.295 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2023-10-18T03:31:15.313 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T03:31:15.319 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T03:31:15.319 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2023-10-18T03:31:15.382 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T03:31:15.416 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T03:31:15.429 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-18T03:31:15.430 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-18T03:31:15.431 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T03:31:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T03:31:15.449 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-10-18T03:31:15.507 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2023-10-18T03:31:15.507 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2023-10-18T03:31:15.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2023-10-18T03:31:17.630 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-10-18T03:31:19.005 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2023-10-18T03:31:20.215 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 7.9 MB 00:00 2023-10-18T03:31:22.547 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 14 MB/s | 18 MB 00:01 2023-10-18T03:31:27.247 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 5.7 MB 00:01 2023-10-18T03:31:29.440 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 14 kB 00:01 2023-10-18T03:31:31.391 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 10 MB/s | 19 MB 00:01 2023-10-18T03:31:36.041 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2023-10-18T03:31:38.147 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-18T03:31:38.148 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-18T03:31:38.240 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:31:38.247 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:31:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:31:38.248 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:31:38.248 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:31:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-376.el9 CentOS-BaseOS 5.3 M 2023-10-18T03:31:38.248 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:31:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-376.el9 CentOS-BaseOS 20 M 2023-10-18T03:31:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-376.el9 CentOS-BaseOS 38 M 2023-10-18T03:31:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-376.el9 CentOS-BaseOS 32 M 2023-10-18T03:31:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:31:38.250 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:31:38.250 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:31:38.250 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2023-10-18T03:31:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:31:38.251 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 95 M 2023-10-18T03:31:38.251 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 120 M 2023-10-18T03:31:38.251 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:31:38.558 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-376.el9.x86_64.rpm 17 MB/s | 5.3 MB 00:00 2023-10-18T03:31:39.017 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-376.el9.x86_64.rpm 26 MB/s | 20 MB 00:00 2023-10-18T03:31:39.593 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-core-5.14.0-376.el9.x86_6 31 MB/s | 32 MB 00:01 2023-10-18T03:31:40.068 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-5.14.0-376.el9.x86_64.rpm 21 MB/s | 38 MB 00:01 2023-10-18T03:31:40.071 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:31:40.071 INFO:teuthology.orchestra.run.smithi003.stdout:Total 52 MB/s | 95 MB 00:01 2023-10-18T03:31:40.072 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:31:40.522 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:31:40.522 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:31:41.531 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:31:41.532 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:31:44.005 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:31:44.649 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-376.el9.x86_64 1/4 2023-10-18T03:31:44.778 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-18T03:31:47.643 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-18T03:31:47.830 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-18T03:31:52.765 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-18T03:31:52.893 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-376.el9.x86_64 4/4 2023-10-18T03:31:57.861 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-376.el9.x86_64 4/4 2023-10-18T03:32:28.089 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-376.el9.x86_64 4/4 2023-10-18T03:32:28.133 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-376.el9.x86_64 4/4 2023-10-18T03:32:28.569 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-376.el9.x86_64 4/4 2023-10-18T03:32:28.570 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-376.el9.x86_64 1/4 2023-10-18T03:32:28.570 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-376.el9.x86_64 2/4 2023-10-18T03:32:28.570 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-376.el9.x86_64 3/4 2023-10-18T03:32:28.970 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-376.el9.x86_64 4/4 2023-10-18T03:32:28.971 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:32:28.971 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:32:28.971 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-376.el9.x86_64 2023-10-18T03:32:28.971 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-376.el9.x86_64 2023-10-18T03:32:28.972 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-376.el9.x86_64 2023-10-18T03:32:28.972 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-376.el9.x86_64 2023-10-18T03:32:28.972 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:32:28.972 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:32:29.457 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T03:32:29.608 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-376.el9.x86_64 2023-10-18T03:32:29.609 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-376.el9.x86_64 2023-10-18T03:32:29.609 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-376.el9.x86_64 2023-10-18T03:32:29.609 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T03:32:29.609 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T03:32:29.610 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-18T03:32:29.610 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T03:32:29.610 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T03:32:29.719 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-376.el9.x86_64 2023-10-18T03:32:29.720 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-376.el9.x86_64 2023-10-18T03:32:29.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2023-10-18T03:32:29.762 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2023-10-18T03:32:29.762 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2023-10-18T03:32:29.762 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2023-10-18T03:32:29.762 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2023-10-18T03:32:29.763 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2023-10-18T03:32:29.763 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-18T03:32:29.763 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2023-10-18T03:32:29.763 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2023-10-18T03:32:29.764 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2023-10-18T03:32:29.764 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-18T03:32:29.764 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-18T03:32:29.764 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-18T03:32:29.764 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-18T03:32:29.765 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2023-10-18T03:32:29.765 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2023-10-18T03:32:29.765 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T03:32:29.765 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2023-10-18T03:32:29.765 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2023-10-18T03:32:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:32:29.766 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T03:32:29.766 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T03:32:29.766 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T03:32:29.767 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2023-10-18T03:32:29.767 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:32:29.767 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T03:32:29.768 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T03:32:29.768 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-376.el9.x86_64 2023-10-18T03:32:29.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T03:32:30.436 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2023-10-18T03:32:31.896 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-18T03:32:31.916 INFO:teuthology.orchestra.run.smithi003.stderr:done 2023-10-18T03:32:31.919 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-10-18T03:32:31.935 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.MtZX4gBrKg 2023-10-18T03:32:31.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MtZX4gBrKg 2023-10-18T03:32:32.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.MtZX4gBrKg 2023-10-18T03:32:32.154 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.MtZX4gBrKg is 6KB 2023-10-18T03:32:32.202 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.MtZX4gBrKg 2023-10-18T03:32:32.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T03:32:32.295 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-18T03:32:32.296 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-18T03:32:32.296 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-18T03:32:32.296 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-376.el9.x86_64.conf 2023-10-18T03:32:32.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-376.el9.x86_64 2023-10-18T03:32:32.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2023-10-18T03:33:02.476 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-18T03:33:02.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T03:33:02.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T03:33:20.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-10-18T03:33:29.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T03:33:29.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T03:34:29.958 DEBUG:teuthology.orchestra.remote:timed out 2023-10-18T03:34:41.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T03:34:41.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T03:34:42.249 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-10-18T03:34:42.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T03:34:42.625 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-376.el9.x86_64"... 2023-10-18T03:34:42.625 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-10-18T03:34:42.641 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-376.el9.x86_64 2023-10-18T03:34:42.641 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-376.el9.x86_64 vs 5.14.0-376.el9.x86_64 2023-10-18T03:34:42.641 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T03:34:42.641 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T03:34:43.643 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-18T03:34:43.643 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T03:34:43.714 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2023-10-18T03:34:43.755 DEBUG:teuthology.parallel:result is None 2023-10-18T03:34:43.755 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T03:34:43.764 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T03:34:43.764 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T03:34:43.782 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T03:34:43.789 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T03:34:43.799 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T03:34:43.800 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T03:34:43.850 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T03:34:43.857 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T03:34:43.858 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T03:34:43.927 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T03:34:43.944 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T03:34:43.945 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T03:34:43.952 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T03:34:43.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T03:34:44.018 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T03:34:44.027 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T03:34:44.028 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T03:34:44.074 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T03:34:44.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T03:34:44.212 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T03:34:44.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T03:34:44.342 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:34:44.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T03:34:44.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2023-10-18T03:34:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T03:34:44.681 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T03:34:44.690 INFO:teuthology.task.internal:Starting timer... 2023-10-18T03:34:44.690 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T03:34:44.738 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T03:34:44.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2023-10-18T03:34:44.794 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2023-10-18T03:34:44.796 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T03:34:44.797 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2023-10-18T03:34:44.835 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2023-10-18T03:34:44.835 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive'} 2023-10-18T03:34:44.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T03:34:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T03:34:44.908 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T03:34:44.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2023-10-18T03:34:45.013 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T03:34:45.022 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-18T03:34:45.023 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T03:34:45.038 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T03:34:45.039 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T03:40:16.614 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com')] 2023-10-18T03:40:16.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T03:40:16.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T03:40:16.717 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-10-18T03:40:16.788 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-10-18T03:40:16.789 INFO:teuthology.run_tasks:Running task clock... 2023-10-18T03:40:16.800 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-18T03:40:16.800 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-18T03:40:16.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T03:40:16.863 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-18T03:40:16.883 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-18T03:40:16.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2023-10-18T03:40:16.937 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2023-10-18T03:40:16.958 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-18T03:40:16.978 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-18T03:40:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2023-10-18T03:40:17.029 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T03:40:17.029 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-10-18T03:40:17.029 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T03:40:17.029 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T03:40:17.030 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T03:40:17.030 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T03:40:17.030 INFO:teuthology.run_tasks:Running task install... 2023-10-18T03:40:17.039 DEBUG:teuthology.task.install:project ceph 2023-10-18T03:40:17.040 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95'}} 2023-10-18T03:40:17.040 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95'} 2023-10-18T03:40:17.040 INFO:teuthology.task.install:Using flavor: crimson 2023-10-18T03:40:17.046 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-18T03:40:17.047 INFO:teuthology.task.install:extra packages: [] 2023-10-18T03:40:17.047 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95', 'tag': None, 'wait_for_package': False} 2023-10-18T03:40:17.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=fbf795c854382dc4ab29c95372096a3ba925ed95 2023-10-18T03:40:17.271 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-17/fbf795c854382dc4ab29c95372096a3ba925ed95/centos/9/flavors/crimson/ 2023-10-18T03:40:17.272 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6794.gfbf795c8 2023-10-18T03:40:17.443 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-17/fbf795c854382dc4ab29c95372096a3ba925ed95/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-17/fbf795c854382dc4ab29c95372096a3ba925ed95/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-17/fbf795c854382dc4ab29c95372096a3ba925ed95/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-18T03:40:17.444 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:40:17.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-18T03:40:17.477 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-18T03:40:17.477 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fbf795c854382dc4ab29c95372096a3ba925ed95/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-18T03:40:17.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-18T03:40:17.636 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-18T03:40:17.666 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2023-10-18T03:40:17.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2023-10-18T03:40:17.913 INFO:teuthology.orchestra.run.smithi003.stdout:72 files removed 2023-10-18T03:40:17.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2023-10-18T03:40:18.622 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 234 kB/s | 81 kB 00:00 2023-10-18T03:40:19.056 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-10-18T03:40:19.465 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2023-10-18T03:40:19.859 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-10-18T03:40:21.334 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2023-10-18T03:40:22.850 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 7.9 MB 00:01 2023-10-18T03:40:25.670 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-10-18T03:40:30.236 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 5.7 MB 00:01 2023-10-18T03:40:31.611 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 14 kB 00:00 2023-10-18T03:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 16 MB/s | 19 MB 00:01 2023-10-18T03:40:37.579 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2023-10-18T03:40:39.966 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:40:39.970 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:40:39.970 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:40:39.970 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:40:39.970 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:40:39.970 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 8.4 M 2023-10-18T03:40:39.971 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2023-10-18T03:40:39.971 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 3.1 M 2023-10-18T03:40:39.971 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 3.0 M 2023-10-18T03:40:39.971 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:40:39.972 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options 2023-10-18T03:40:39.972 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-18T03:40:39.972 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 6.6 M 2023-10-18T03:40:39.972 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 17 M 2023-10-18T03:40:39.972 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 24 k 2023-10-18T03:40:39.973 INFO:teuthology.orchestra.run.smithi003.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-18T03:40:39.973 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-18T03:40:39.973 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-18T03:40:39.973 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-18T03:40:39.973 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 727 k 2023-10-18T03:40:39.973 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-18T03:40:39.974 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-18T03:40:39.974 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-18T03:40:39.974 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 469 k 2023-10-18T03:40:39.974 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-18T03:40:39.975 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 4.5 M 2023-10-18T03:40:39.975 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-18T03:40:39.975 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-18T03:40:39.975 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-18T03:40:39.976 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse 2023-10-18T03:40:39.976 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 45 k 2023-10-18T03:40:39.976 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 127 k 2023-10-18T03:40:39.976 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 161 k 2023-10-18T03:40:39.976 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-18T03:40:39.977 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 321 k 2023-10-18T03:40:39.977 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 298 k 2023-10-18T03:40:39.977 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 100 k 2023-10-18T03:40:39.977 INFO:teuthology.orchestra.run.smithi003.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-18T03:40:39.977 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-18T03:40:39.978 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:40:39.978 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:40:39.978 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:40:39.978 INFO:teuthology.orchestra.run.smithi003.stdout:Install 28 Packages 2023-10-18T03:40:39.978 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2023-10-18T03:40:39.979 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:40:39.979 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 54 M 2023-10-18T03:40:39.979 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:40:40.864 INFO:teuthology.orchestra.run.smithi003.stdout:(1/30): ceph-base-18.0.0-6794.gfbf795c8.el9.x86 9.7 MB/s | 6.6 MB 00:00 2023-10-18T03:40:40.915 INFO:teuthology.orchestra.run.smithi003.stdout:(2/30): ceph-selinux-18.0.0-6794.gfbf795c8.el9. 482 kB/s | 24 kB 00:00 2023-10-18T03:40:40.998 INFO:teuthology.orchestra.run.smithi003.stdout:(3/30): libcephfs2-18.0.0-6794.gfbf795c8.el9.x8 8.5 MB/s | 727 kB 00:00 2023-10-18T03:40:41.107 INFO:teuthology.orchestra.run.smithi003.stdout:(4/30): ceph-radosgw-18.0.0-6794.gfbf795c8.el9. 9.1 MB/s | 8.4 MB 00:00 2023-10-18T03:40:41.132 INFO:teuthology.orchestra.run.smithi003.stdout:(5/30): libradosstriper1-18.0.0-6794.gfbf795c8. 3.4 MB/s | 469 kB 00:00 2023-10-18T03:40:41.182 INFO:teuthology.orchestra.run.smithi003.stdout:(6/30): python3-ceph-argparse-18.0.0-6794.gfbf7 906 kB/s | 45 kB 00:00 2023-10-18T03:40:41.249 INFO:teuthology.orchestra.run.smithi003.stdout:(7/30): python3-ceph-common-18.0.0-6794.gfbf795 1.9 MB/s | 127 kB 00:00 2023-10-18T03:40:41.324 INFO:teuthology.orchestra.run.smithi003.stdout:(8/30): python3-cephfs-18.0.0-6794.gfbf795c8.el 2.1 MB/s | 161 kB 00:00 2023-10-18T03:40:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:(9/30): librgw2-18.0.0-6794.gfbf795c8.el9.x86_6 16 MB/s | 4.5 MB 00:00 2023-10-18T03:40:41.408 INFO:teuthology.orchestra.run.smithi003.stdout:(10/30): python3-rados-18.0.0-6794.gfbf795c8.el 3.7 MB/s | 321 kB 00:00 2023-10-18T03:40:41.434 INFO:teuthology.orchestra.run.smithi003.stdout:(11/30): python3-rbd-18.0.0-6794.gfbf795c8.el9. 5.8 MB/s | 298 kB 00:00 2023-10-18T03:40:41.459 INFO:teuthology.orchestra.run.smithi003.stdout:(12/30): python3-rgw-18.0.0-6794.gfbf795c8.el9. 1.9 MB/s | 100 kB 00:00 2023-10-18T03:40:41.584 INFO:teuthology.orchestra.run.smithi003.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2023-10-18T03:40:41.626 INFO:teuthology.orchestra.run.smithi003.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 954 kB/s | 159 kB 00:00 2023-10-18T03:40:41.693 INFO:teuthology.orchestra.run.smithi003.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2023-10-18T03:40:41.718 INFO:teuthology.orchestra.run.smithi003.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2023-10-18T03:40:41.743 INFO:teuthology.orchestra.run.smithi003.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2023-10-18T03:40:41.852 INFO:teuthology.orchestra.run.smithi003.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2023-10-18T03:40:42.127 INFO:teuthology.orchestra.run.smithi003.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 289 kB/s | 111 kB 00:00 2023-10-18T03:40:42.236 INFO:teuthology.orchestra.run.smithi003.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 861 kB/s | 309 kB 00:00 2023-10-18T03:40:42.353 INFO:teuthology.orchestra.run.smithi003.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 209 kB/s | 24 kB 00:00 2023-10-18T03:40:42.470 INFO:teuthology.orchestra.run.smithi003.stdout:(22/30): libarrow-9.0.0-9.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2023-10-18T03:40:42.495 INFO:teuthology.orchestra.run.smithi003.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 345 kB/s | 49 kB 00:00 2023-10-18T03:40:42.537 INFO:teuthology.orchestra.run.smithi003.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2023-10-18T03:40:42.587 INFO:teuthology.orchestra.run.smithi003.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 8.9 MB/s | 837 kB 00:00 2023-10-18T03:40:42.613 INFO:teuthology.orchestra.run.smithi003.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2023-10-18T03:40:42.780 INFO:teuthology.orchestra.run.smithi003.stdout:(27/30): ceph-common-18.0.0-6794.gfbf795c8.el9. 6.6 MB/s | 17 MB 00:02 2023-10-18T03:40:42.813 INFO:teuthology.orchestra.run.smithi003.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2023-10-18T03:40:42.955 INFO:teuthology.orchestra.run.smithi003.stdout:(29/30): librados2-18.0.0-6794.gfbf795c8.el9.x8 9.1 MB/s | 3.1 MB 00:00 2023-10-18T03:40:43.055 INFO:teuthology.orchestra.run.smithi003.stdout:(30/30): librbd1-18.0.0-6794.gfbf795c8.el9.x86_ 11 MB/s | 3.0 MB 00:00 2023-10-18T03:40:43.061 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:40:43.061 INFO:teuthology.orchestra.run.smithi003.stdout:Total 17 MB/s | 54 MB 00:03 2023-10-18T03:40:43.073 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:40:43.111 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:40:43.111 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:40:43.609 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:40:43.610 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:40:45.021 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:40:45.403 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-18T03:40:45.458 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:18.0.0-6794.gfbf795c8.el9.x86_64 2/32 2023-10-18T03:40:45.550 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:18.0.0-6794.gfbf795c8.el9.x86_64 2/32 2023-10-18T03:40:45.641 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:18.0.0-6794.gfbf795c8.el9.x86_64 3/32 2023-10-18T03:40:45.667 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:18.0.0-6794.gfbf795c8.el9.x86_64 4/32 2023-10-18T03:40:45.730 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:18.0.0-6794.gfbf795c8.el9.x86_64 4/32 2023-10-18T03:40:45.773 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-18T03:40:45.805 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-18T03:40:45.835 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-18T03:40:45.879 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:18.0.0-6794.gfbf795c8.el9. 8/32 2023-10-18T03:40:45.948 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:18.0.0-6794.gfbf795c8.el9.x86_64 9/32 2023-10-18T03:40:45.971 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:18.0.0-6794.gfbf795c8.el9.x86_6 10/32 2023-10-18T03:40:46.042 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6794.gfbf795c8.el9.x86_6 10/32 2023-10-18T03:40:46.078 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-18T03:40:46.193 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-18T03:40:46.227 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-18T03:40:46.268 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-18T03:40:46.299 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-18T03:40:46.335 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-18T03:40:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-18T03:40:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-18T03:40:46.733 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-18T03:40:46.769 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:18.0.0-6794.gfbf795c8.el9.x86_64 20/32 2023-10-18T03:40:46.866 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:18.0.0-6794.gfbf795c8.el9.x86_64 20/32 2023-10-18T03:40:46.901 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:18.0.0-6794.gfbf795c8.el9.x86_64 21/32 2023-10-18T03:40:46.942 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-18T03:40:47.395 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-18T03:40:47.428 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:18.0.0-6794.gfbf795c8.el9.x86_64 24/32 2023-10-18T03:40:47.499 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:18.0.0-6794.gfbf795c8.el9.x86_64 24/32 2023-10-18T03:40:47.578 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:18.0.0-6794.gfbf795c8.el9.x86_64 25/32 2023-10-18T03:40:47.622 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:18.0.0-6794.gfbf795c8.el9.x8 26/32 2023-10-18T03:40:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-6794.gfbf795c8.el9.x86_64 27/32 2023-10-18T03:40:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:18.0.0-6794.gfbf795c8.el9.x86_64 27/32 2023-10-18T03:40:50.517 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:18.0.0-6794.gfbf795c8.el9.x86_64 27/32 2023-10-18T03:40:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:18.0.0-6794.gfbf795c8.el9.x86_64 28/32 2023-10-18T03:40:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:18.0.0-6794.gfbf795c8.el9.x86_64 28/32 2023-10-18T03:40:50.635 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-18T03:40:50.635 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-18T03:40:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:40:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:18.0.0-6794.gfbf795c8.el9.x86_64 29/32 2023-10-18T03:41:00.965 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6794.gfbf795c8.el9.x86_64 29/32 2023-10-18T03:41:00.966 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2023-10-18T03:41:00.966 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2023-10-18T03:41:00.966 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2023-10-18T03:41:00.966 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2023-10-18T03:41:00.967 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2023-10-18T03:41:00.967 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2023-10-18T03:41:00.967 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2023-10-18T03:41:00.967 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:01.832 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:18.0.0-6794.gfbf795c8.el9.x86_64 30/32 2023-10-18T03:41:01.858 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6794.gfbf795c8.el9.x86_64 30/32 2023-10-18T03:41:01.858 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-18T03:41:01.858 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-18T03:41:01.858 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-18T03:41:01.859 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-18T03:41:01.859 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:01.860 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-18T03:41:01.926 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-18T03:41:01.927 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-18T03:41:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-18T03:41:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/32 2023-10-18T03:41:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:18.0.0-6794.gfbf795c8.el9.x86_64 2/32 2023-10-18T03:41:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:18.0.0-6794.gfbf795c8.el9.x86_64 3/32 2023-10-18T03:41:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:18.0.0-6794.gfbf795c8.el9.x86_64 4/32 2023-10-18T03:41:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:18.0.0-6794.gfbf795c8.el9.x86_64 5/32 2023-10-18T03:41:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:18.0.0-6794.gfbf795c8.el9.x86_6 6/32 2023-10-18T03:41:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:18.0.0-6794.gfbf795c8.el9.x86_64 7/32 2023-10-18T03:41:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6794.gfbf795c8.el9. 8/32 2023-10-18T03:41:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:18.0.0-6794.gfbf795c8.el9.x8 9/32 2023-10-18T03:41:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:18.0.0-6794.gfbf795c8.el9.x86_64 10/32 2023-10-18T03:41:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:18.0.0-6794.gfbf795c8.el9.x86_64 11/32 2023-10-18T03:41:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:18.0.0-6794.gfbf795c8.el9.x86_64 12/32 2023-10-18T03:41:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:18.0.0-6794.gfbf795c8.el9.x86_64 13/32 2023-10-18T03:41:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-18T03:41:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-18T03:41:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-18T03:41:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-18T03:41:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-18T03:41:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-18T03:41:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-18T03:41:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-18T03:41:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-18T03:41:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-18T03:41:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-18T03:41:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-18T03:41:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-18T03:41:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-18T03:41:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-18T03:41:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:18.0.0-6794.gfbf795c8.el9.x86_64 29/32 2023-10-18T03:41:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-18T03:41:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:18.0.0-6794.gfbf795c8.el9.x86_64 31/32 2023-10-18T03:41:04.025 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-18T03:41:04.025 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:04.026 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2023-10-18T03:41:04.026 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.026 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.026 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:41:04.027 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-18T03:41:04.027 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.027 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.027 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.027 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.028 INFO:teuthology.orchestra.run.smithi003.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-18T03:41:04.028 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-18T03:41:04.028 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-18T03:41:04.028 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-18T03:41:04.028 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.029 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-18T03:41:04.029 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-18T03:41:04.029 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-18T03:41:04.029 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.029 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-18T03:41:04.029 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.030 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-18T03:41:04.030 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-18T03:41:04.030 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-18T03:41:04.030 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.030 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-18T03:41:04.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:04.032 INFO:teuthology.orchestra.run.smithi003.stdout: re2-1:20211101-3.el9.x86_64 2023-10-18T03:41:04.032 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-18T03:41:04.032 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:04.032 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:41:04.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2023-10-18T03:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:27 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:41:05.145 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:41:05.146 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:41:05.146 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:41:05.147 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:41:05.147 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:41:05.147 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 117 M 2023-10-18T03:41:05.147 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:41:05.147 INFO:teuthology.orchestra.run.smithi003.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-18T03:41:05.148 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-18T03:41:05.148 INFO:teuthology.orchestra.run.smithi003.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-18T03:41:05.148 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 161 k 2023-10-18T03:41:05.148 INFO:teuthology.orchestra.run.smithi003.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-18T03:41:05.148 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-18T03:41:05.149 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-18T03:41:05.149 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-18T03:41:05.149 INFO:teuthology.orchestra.run.smithi003.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-18T03:41:05.149 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:05.149 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:41:05.149 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:41:05.150 INFO:teuthology.orchestra.run.smithi003.stdout:Install 10 Packages 2023-10-18T03:41:05.150 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:05.151 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 120 M 2023-10-18T03:41:05.151 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.1 G 2023-10-18T03:41:05.151 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:41:05.438 INFO:teuthology.orchestra.run.smithi003.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.5 MB/s | 418 kB 00:00 2023-10-18T03:41:05.472 INFO:teuthology.orchestra.run.smithi003.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.5 MB/s | 186 kB 00:00 2023-10-18T03:41:05.514 INFO:teuthology.orchestra.run.smithi003.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-10-18T03:41:05.556 INFO:teuthology.orchestra.run.smithi003.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-10-18T03:41:05.581 INFO:teuthology.orchestra.run.smithi003.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2023-10-18T03:41:05.640 INFO:teuthology.orchestra.run.smithi003.stdout:(6/10): libcephsqlite-18.0.0-6794.gfbf795c8.el9 444 kB/s | 161 kB 00:00 2023-10-18T03:41:05.907 INFO:teuthology.orchestra.run.smithi003.stdout:(7/10): cryptopp-8.6.0-1.el9.x86_64.rpm 4.7 MB/s | 1.3 MB 00:00 2023-10-18T03:41:05.948 INFO:teuthology.orchestra.run.smithi003.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.9 MB/s | 122 kB 00:00 2023-10-18T03:41:06.099 INFO:teuthology.orchestra.run.smithi003.stdout:(9/10): jq-1.6-15.el9.x86_64.rpm 361 kB/s | 187 kB 00:00 2023-10-18T03:41:10.757 INFO:teuthology.orchestra.run.smithi003.stdout:(10/10): ceph-test-18.0.0-6794.gfbf795c8.el9.x8 21 MB/s | 117 MB 00:05 2023-10-18T03:41:10.759 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:41:10.760 INFO:teuthology.orchestra.run.smithi003.stdout:Total 21 MB/s | 120 MB 00:05 2023-10-18T03:41:10.775 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:41:10.800 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:41:10.800 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:41:11.661 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:41:11.662 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:41:12.810 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:41:12.912 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-18T03:41:12.949 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-18T03:41:13.001 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-18T03:41:13.052 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-18T03:41:13.083 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-18T03:41:13.114 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-18T03:41:13.159 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-18T03:41:13.202 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-18T03:41:13.228 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:18.0.0-6794.gfbf795c8.el9.x86_64 9/10 2023-10-18T03:41:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6794.gfbf795c8.el9.x86_64 9/10 2023-10-18T03:41:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:18.0.0-6794.gfbf795c8.el9.x86_64 10/10 2023-10-18T03:41:38.559 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:18.0.0-6794.gfbf795c8.el9.x86_64 10/10 2023-10-18T03:41:38.560 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/10 2023-10-18T03:41:38.560 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:18.0.0-6794.gfbf795c8.el9.x86_64 2/10 2023-10-18T03:41:38.560 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-18T03:41:38.560 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-18T03:41:38.561 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-18T03:41:38.561 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-18T03:41:38.561 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-18T03:41:38.561 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-18T03:41:38.561 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-18T03:41:39.007 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-18T03:41:39.008 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:39.008 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:41:39.008 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:39.008 INFO:teuthology.orchestra.run.smithi003.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-18T03:41:39.009 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-15.el9.x86_64 2023-10-18T03:41:39.009 INFO:teuthology.orchestra.run.smithi003.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-18T03:41:39.009 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:41:39.009 INFO:teuthology.orchestra.run.smithi003.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-18T03:41:39.009 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-18T03:41:39.010 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-18T03:41:39.010 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-18T03:41:39.010 INFO:teuthology.orchestra.run.smithi003.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-18T03:41:39.010 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:39.010 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:41:39.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2023-10-18T03:41:39.925 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:02 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:41:40.073 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:41:40.075 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2023-10-18T03:41:40.075 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:41:40.076 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2023-10-18T03:41:40.076 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:41:40.076 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 6.5 k 2023-10-18T03:41:40.076 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:41:40.077 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 2.0 M 2023-10-18T03:41:40.077 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 1.5 M 2023-10-18T03:41:40.077 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 242 k 2023-10-18T03:41:40.077 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 8.8 M 2023-10-18T03:41:40.077 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 118 M 2023-10-18T03:41:40.078 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-18T03:41:40.078 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-18T03:41:40.078 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-18T03:41:40.078 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-18T03:41:40.078 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-18T03:41:40.079 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-18T03:41:40.079 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-18T03:41:40.079 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-18T03:41:40.079 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-18T03:41:40.079 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-18T03:41:40.080 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-18T03:41:40.080 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-18T03:41:40.080 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-18T03:41:40.080 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-18T03:41:40.080 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-18T03:41:40.081 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-18T03:41:40.081 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-18T03:41:40.081 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-18T03:41:40.081 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-18T03:41:40.081 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-18T03:41:40.082 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-18T03:41:40.082 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-18T03:41:40.082 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-18T03:41:40.082 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-18T03:41:40.083 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-18T03:41:40.083 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-18T03:41:40.083 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-18T03:41:40.083 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:40.083 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:41:40.084 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2023-10-18T03:41:40.084 INFO:teuthology.orchestra.run.smithi003.stdout:Install 33 Packages 2023-10-18T03:41:40.084 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:41:40.085 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 135 M 2023-10-18T03:41:40.085 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.4 G 2023-10-18T03:41:40.085 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:41:41.016 INFO:teuthology.orchestra.run.smithi003.stdout:(1/33): ceph-18.0.0-6794.gfbf795c8.el9.x86_64.r 26 kB/s | 6.5 kB 00:00 2023-10-18T03:41:41.975 INFO:teuthology.orchestra.run.smithi003.stdout:(2/33): ceph-mgr-18.0.0-6794.gfbf795c8.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2023-10-18T03:41:42.192 INFO:teuthology.orchestra.run.smithi003.stdout:(3/33): ceph-mds-18.0.0-6794.gfbf795c8.el9.x86_ 1.4 MB/s | 2.0 MB 00:01 2023-10-18T03:41:42.342 INFO:teuthology.orchestra.run.smithi003.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6794.gfbf7 1.6 MB/s | 242 kB 00:00 2023-10-18T03:41:42.443 INFO:teuthology.orchestra.run.smithi003.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-10-18T03:41:42.476 INFO:teuthology.orchestra.run.smithi003.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2023-10-18T03:41:42.518 INFO:teuthology.orchestra.run.smithi003.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 930 kB/s | 39 kB 00:00 2023-10-18T03:41:42.568 INFO:teuthology.orchestra.run.smithi003.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-10-18T03:41:42.602 INFO:teuthology.orchestra.run.smithi003.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-18T03:41:42.644 INFO:teuthology.orchestra.run.smithi003.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-10-18T03:41:42.677 INFO:teuthology.orchestra.run.smithi003.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-18T03:41:42.720 INFO:teuthology.orchestra.run.smithi003.stdout:(12/33): python3-urllib3-1.26.5-3.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2023-10-18T03:41:43.486 INFO:teuthology.orchestra.run.smithi003.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 1.6 MB/s | 1.2 MB 00:00 2023-10-18T03:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 472 kB/s | 51 kB 00:00 2023-10-18T03:41:43.754 INFO:teuthology.orchestra.run.smithi003.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 275 kB/s | 43 kB 00:00 2023-10-18T03:41:43.895 INFO:teuthology.orchestra.run.smithi003.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2023-10-18T03:41:44.004 INFO:teuthology.orchestra.run.smithi003.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2023-10-18T03:41:44.063 INFO:teuthology.orchestra.run.smithi003.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2023-10-18T03:41:44.121 INFO:teuthology.orchestra.run.smithi003.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2023-10-18T03:41:44.180 INFO:teuthology.orchestra.run.smithi003.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2023-10-18T03:41:44.238 INFO:teuthology.orchestra.run.smithi003.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 335 kB/s | 19 kB 00:00 2023-10-18T03:41:44.297 INFO:teuthology.orchestra.run.smithi003.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2023-10-18T03:41:44.356 INFO:teuthology.orchestra.run.smithi003.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 792 kB/s | 46 kB 00:00 2023-10-18T03:41:44.406 INFO:teuthology.orchestra.run.smithi003.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2023-10-18T03:41:44.482 INFO:teuthology.orchestra.run.smithi003.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2023-10-18T03:41:44.540 INFO:teuthology.orchestra.run.smithi003.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2023-10-18T03:41:44.665 INFO:teuthology.orchestra.run.smithi003.stdout:(27/33): ceph-mon-18.0.0-6794.gfbf795c8.el9.x86 2.4 MB/s | 8.8 MB 00:03 2023-10-18T03:41:44.690 INFO:teuthology.orchestra.run.smithi003.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 599 kB/s | 90 kB 00:00 2023-10-18T03:41:44.749 INFO:teuthology.orchestra.run.smithi003.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2023-10-18T03:41:44.799 INFO:teuthology.orchestra.run.smithi003.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 268 kB/s | 36 kB 00:00 2023-10-18T03:41:44.825 INFO:teuthology.orchestra.run.smithi003.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2023-10-18T03:41:44.866 INFO:teuthology.orchestra.run.smithi003.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2023-10-18T03:41:54.041 INFO:teuthology.orchestra.run.smithi003.stdout:(33/33): ceph-osd-18.0.0-6794.gfbf795c8.el9.x86 9.8 MB/s | 118 MB 00:12 2023-10-18T03:41:54.044 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:41:54.045 INFO:teuthology.orchestra.run.smithi003.stdout:Total 9.7 MB/s | 135 MB 00:13 2023-10-18T03:41:54.078 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:41:54.121 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:41:54.121 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:41:55.137 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:41:55.138 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:41:57.016 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:41:57.085 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-18T03:41:57.117 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-18T03:41:57.150 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-18T03:41:57.244 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-18T03:41:57.336 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-18T03:41:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-18T03:41:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-18T03:41:57.502 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-18T03:41:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-10-18T03:41:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-18T03:41:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-18T03:41:57.705 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-18T03:41:57.914 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-18T03:41:57.961 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-18T03:41:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-18T03:41:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-18T03:41:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-18T03:41:58.144 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-18T03:41:58.192 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-18T03:41:58.225 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-18T03:41:58.267 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-18T03:41:58.300 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-18T03:41:58.393 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-18T03:41:58.432 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-18T03:41:58.507 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-18T03:41:58.648 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-18T03:41:58.779 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-18T03:41:58.947 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6794.gfbf795c8.el9. 28/33 2023-10-18T03:41:58.969 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:18.0.0-6794.gfbf795c8.el9.x86_64 29/33 2023-10-18T03:41:58.995 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6794.gfbf795c8.el9.x86_64 29/33 2023-10-18T03:41:58.995 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-18T03:41:58.996 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-18T03:41:58.996 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-18T03:41:58.996 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-18T03:41:58.996 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:18.593 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:18.0.0-6794.gfbf795c8.el9.x86_64 30/33 2023-10-18T03:42:18.621 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:18.0.0-6794.gfbf795c8.el9.x86_64 30/33 2023-10-18T03:42:18.621 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-18T03:42:18.621 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-18T03:42:18.621 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-18T03:42:18.622 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-18T03:42:18.622 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:19.561 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:18.0.0-6794.gfbf795c8.el9.x86_64 31/33 2023-10-18T03:42:19.586 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:18.0.0-6794.gfbf795c8.el9.x86_64 31/33 2023-10-18T03:42:19.587 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-18T03:42:19.587 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-18T03:42:19.587 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-18T03:42:19.587 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-18T03:42:19.587 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:19.803 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:18.0.0-6794.gfbf795c8.el9.x86_64 32/33 2023-10-18T03:42:19.828 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:18.0.0-6794.gfbf795c8.el9.x86_64 32/33 2023-10-18T03:42:19.828 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-18T03:42:19.828 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-18T03:42:19.829 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-18T03:42:19.829 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-18T03:42:19.829 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:19.914 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:18.0.0-6794.gfbf795c8.el9.x86_64 33/33 2023-10-18T03:42:30.016 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:18.0.0-6794.gfbf795c8.el9.x86_64 33/33 2023-10-18T03:42:30.017 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/33 2023-10-18T03:42:30.017 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:18.0.0-6794.gfbf795c8.el9.x86_64 2/33 2023-10-18T03:42:30.017 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:18.0.0-6794.gfbf795c8.el9.x86_64 3/33 2023-10-18T03:42:30.017 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:18.0.0-6794.gfbf795c8.el9.x86_64 4/33 2023-10-18T03:42:30.018 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:18.0.0-6794.gfbf795c8.el9.x86_64 5/33 2023-10-18T03:42:30.018 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6794.gfbf795c8.el9. 6/33 2023-10-18T03:42:30.018 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-18T03:42:30.018 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-18T03:42:30.018 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-18T03:42:30.019 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-18T03:42:30.019 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-18T03:42:30.019 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-18T03:42:30.019 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-18T03:42:30.019 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-10-18T03:42:30.020 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-18T03:42:30.020 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-18T03:42:30.020 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-18T03:42:30.020 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-18T03:42:30.020 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-18T03:42:30.021 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-18T03:42:30.021 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-18T03:42:30.021 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-18T03:42:30.021 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-18T03:42:30.021 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-18T03:42:30.022 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-18T03:42:30.022 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-18T03:42:30.022 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-18T03:42:30.023 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-18T03:42:30.023 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-18T03:42:30.023 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-18T03:42:30.023 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-18T03:42:30.024 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-18T03:42:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-18T03:42:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:31.028 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:42:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:42:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:42:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:42:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:42:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:42:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:42:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-18T03:42:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-18T03:42:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-18T03:42:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-18T03:42:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-18T03:42:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-18T03:42:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-18T03:42:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-18T03:42:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-18T03:42:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-18T03:42:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-18T03:42:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-18T03:42:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-18T03:42:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-18T03:42:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-18T03:42:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2023-10-18T03:42:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-18T03:42:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-18T03:42:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-18T03:42:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-18T03:42:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-18T03:42:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-18T03:42:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-18T03:42:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-18T03:42:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-18T03:42:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-18T03:42:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-18T03:42:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:31.035 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:42:31.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2023-10-18T03:42:31.879 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:54 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:42:31.971 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:42:32.022 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:42:32.023 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:42:32.023 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:42:32.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2023-10-18T03:42:32.699 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:55 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:42:32.843 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:42:32.843 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:32.844 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:42:32.844 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:32.844 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:42:32.844 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 270 k 2023-10-18T03:42:32.845 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:32.845 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:42:32.845 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:32.845 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-18T03:42:32.845 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:32.846 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 270 k 2023-10-18T03:42:32.846 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 265 k 2023-10-18T03:42:32.846 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:42:33.075 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-18.0.0-6794.gfbf795c8.el9.noarch.rpm 1.1 MB/s | 270 kB 00:00 2023-10-18T03:42:33.076 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:42:33.076 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.1 MB/s | 270 kB 00:00 2023-10-18T03:42:33.077 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:42:33.082 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:42:33.082 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:42:33.085 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:42:33.085 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:42:33.158 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:42:33.557 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 1/1 2023-10-18T03:42:33.658 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 1/1 2023-10-18T03:42:34.052 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 1/1 2023-10-18T03:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 1/1 2023-10-18T03:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:42:34.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2023-10-18T03:42:35.172 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:58 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:42:35.315 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:42:35.316 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:35.316 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-18T03:42:35.317 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:35.317 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:42:35.317 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 144 k 2023-10-18T03:42:35.317 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:35.317 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:42:35.318 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:35.318 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-18T03:42:35.318 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:35.318 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 144 k 2023-10-18T03:42:35.318 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 437 k 2023-10-18T03:42:35.319 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:42:35.539 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-18.0.0-6794.gfbf795 651 kB/s | 144 kB 00:00 2023-10-18T03:42:35.539 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:42:35.540 INFO:teuthology.orchestra.run.smithi003.stdout:Total 646 kB/s | 144 kB 00:00 2023-10-18T03:42:35.540 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:42:35.547 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:42:35.548 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:42:35.602 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:42:35.603 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:42:35.745 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:42:35.790 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6794.gfbf795c8. 1/1 2023-10-18T03:42:35.814 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6794.gfbf795c8. 1/1 2023-10-18T03:42:35.814 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-18T03:42:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-18T03:42:35.815 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:36.789 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6794.gfbf795c8. 1/1 2023-10-18T03:42:36.790 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:36.790 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:42:36.790 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:42:36.790 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:36.791 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:42:37.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2023-10-18T03:42:37.645 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:00 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:42:37.738 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:42:37.790 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:42:37.791 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:42:37.791 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:42:37.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2023-10-18T03:42:38.472 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:01 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:42:38.618 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:42:38.618 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:38.619 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:42:38.619 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:38.619 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:42:38.619 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 1.8 M 2023-10-18T03:42:38.620 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:42:38.620 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 23 k 2023-10-18T03:42:38.620 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 15 k 2023-10-18T03:42:38.620 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-18T03:42:38.620 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-18T03:42:38.620 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-18T03:42:38.621 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2023-10-18T03:42:38.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-18T03:42:38.621 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:38.621 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:42:38.621 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:42:38.622 INFO:teuthology.orchestra.run.smithi003.stdout:Install 7 Packages 2023-10-18T03:42:38.622 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:38.622 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.1 M 2023-10-18T03:42:38.623 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 8.6 M 2023-10-18T03:42:38.623 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:42:38.867 INFO:teuthology.orchestra.run.smithi003.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6794.gfbf 122 kB/s | 23 kB 00:00 2023-10-18T03:42:38.951 INFO:teuthology.orchestra.run.smithi003.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 108 kB/s | 9.0 kB 00:00 2023-10-18T03:42:38.968 INFO:teuthology.orchestra.run.smithi003.stdout:(3/7): ceph-prometheus-alerts-18.0.0-6794.gfbf7 52 kB/s | 15 kB 00:00 2023-10-18T03:42:39.043 INFO:teuthology.orchestra.run.smithi003.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2023-10-18T03:42:39.102 INFO:teuthology.orchestra.run.smithi003.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 230 kB/s | 31 kB 00:00 2023-10-18T03:42:39.177 INFO:teuthology.orchestra.run.smithi003.stdout:(6/7): ceph-mgr-dashboard-18.0.0-6794.gfbf795c8 3.5 MB/s | 1.8 MB 00:00 2023-10-18T03:42:39.202 INFO:teuthology.orchestra.run.smithi003.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2023-10-18T03:42:39.203 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:42:39.204 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.5 MB/s | 2.1 MB 00:00 2023-10-18T03:42:39.204 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:42:39.213 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:42:39.213 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:42:39.280 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:42:39.280 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:42:39.575 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:42:39.628 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-18T03:42:39.662 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-18T03:42:39.719 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-18T03:42:39.757 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-18T03:42:39.786 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6794.gfbf795c8.el9.n 5/7 2023-10-18T03:42:40.034 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6794.gfbf795c8.el9. 6/7 2023-10-18T03:42:40.064 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6794.gfbf795c8.el9.noarc 7/7 2023-10-18T03:42:40.627 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6794.gfbf795c8.el9.noarc 7/7 2023-10-18T03:42:40.628 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6794.gfbf795c8.el9. 1/7 2023-10-18T03:42:40.628 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6794.gfbf795c8.el9.noarc 2/7 2023-10-18T03:42:40.628 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6794.gfbf795c8.el9.n 3/7 2023-10-18T03:42:40.628 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-18T03:42:40.628 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-18T03:42:40.629 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-18T03:42:40.948 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-18T03:42:40.948 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:40.948 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:42:40.949 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:42:40.949 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:42:40.949 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:42:40.949 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-18T03:42:40.949 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-18T03:42:40.949 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-18T03:42:40.950 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-18T03:42:40.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:40.950 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:42:41.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-18T03:42:41.764 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:04 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:42:41.909 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:42:41.910 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2023-10-18T03:42:41.910 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:42:41.910 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2023-10-18T03:42:41.910 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:42:41.911 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 7.4 M 2023-10-18T03:42:41.911 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:42:41.911 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-18T03:42:41.911 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-18T03:42:41.911 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-18T03:42:41.912 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-18T03:42:41.912 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-18T03:42:41.912 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-18T03:42:41.912 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-18T03:42:41.912 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-18T03:42:41.913 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-18T03:42:41.913 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-18T03:42:41.913 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-18T03:42:41.913 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:41.913 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:42:41.914 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2023-10-18T03:42:41.914 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2023-10-18T03:42:41.914 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:41.915 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 37 M 2023-10-18T03:42:41.915 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 209 M 2023-10-18T03:42:41.915 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:42:42.050 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 32 kB 00:00 2023-10-18T03:42:42.075 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 725 kB/s | 17 kB 00:00 2023-10-18T03:42:42.100 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2023-10-18T03:42:42.176 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-10-18T03:42:42.259 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 205 kB 00:00 2023-10-18T03:42:42.326 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2023-10-18T03:42:42.418 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 4.8 MB 00:00 2023-10-18T03:42:42.443 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2023-10-18T03:42:42.760 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2023-10-18T03:42:42.811 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2023-10-18T03:42:42.836 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2023-10-18T03:42:43.578 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 4.4 MB/s | 7.4 MB 00:01 2023-10-18T03:42:43.580 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:42:43.581 INFO:teuthology.orchestra.run.smithi003.stdout:Total 22 MB/s | 37 MB 00:01 2023-10-18T03:42:43.581 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:42:43.605 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:42:43.606 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:42:44.054 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:42:44.054 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:42:44.774 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:42:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-18T03:42:44.899 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-18T03:42:44.998 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-18T03:42:45.036 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-18T03:42:45.220 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-18T03:42:45.251 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-18T03:42:45.334 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-18T03:42:46.030 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-18T03:42:46.145 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-18T03:42:47.473 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-18T03:42:48.552 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-18T03:42:48.583 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6794.gfbf79 12/12 2023-10-18T03:42:51.094 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6794.gfbf79 12/12 2023-10-18T03:42:51.094 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6794.gfbf79 1/12 2023-10-18T03:42:51.094 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-18T03:42:51.095 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-18T03:42:51.095 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-18T03:42:51.095 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-18T03:42:51.095 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-18T03:42:51.095 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-18T03:42:51.096 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-18T03:42:51.096 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-18T03:42:51.096 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-18T03:42:51.096 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-18T03:42:51.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-18T03:42:51.535 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:51.535 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:42:51.535 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:42:51.535 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-18T03:42:51.536 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-18T03:42:51.536 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-18T03:42:51.536 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-18T03:42:51.536 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-18T03:42:51.536 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-18T03:42:51.537 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-18T03:42:51.537 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-18T03:42:51.537 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-18T03:42:51.537 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-18T03:42:51.538 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-18T03:42:51.538 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:51.538 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:42:51.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2023-10-18T03:42:52.485 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:15 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:42:52.631 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:42:52.632 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-18T03:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-18T03:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:42:52.633 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 47 k 2023-10-18T03:42:52.634 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:42:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-18T03:42:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-18T03:42:52.634 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-18T03:42:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-18T03:42:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-18T03:42:52.635 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-18T03:42:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-18T03:42:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-18T03:42:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-18T03:42:52.636 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-18T03:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-18T03:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-18T03:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-18T03:42:52.637 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2023-10-18T03:42:52.638 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:52.638 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.0 M 2023-10-18T03:42:52.639 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2023-10-18T03:42:52.639 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:42:52.968 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 194 kB/s | 24 kB 00:00 2023-10-18T03:42:52.985 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 127 kB/s | 18 kB 00:00 2023-10-18T03:42:53.068 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): ceph-mgr-rook-18.0.0-6794.gfbf795c8.el9 210 kB/s | 47 kB 00:00 2023-10-18T03:42:53.094 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2023-10-18T03:42:53.128 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 928 kB/s | 132 kB 00:00 2023-10-18T03:42:53.228 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 211 kB 00:00 2023-10-18T03:42:53.253 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 269 kB/s | 43 kB 00:00 2023-10-18T03:42:53.428 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2023-10-18T03:42:53.462 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 72 kB/s | 15 kB 00:00 2023-10-18T03:42:53.612 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2023-10-18T03:42:53.687 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-google-auth-2.23.2-1.el9.noarc 454 kB/s | 196 kB 00:00 2023-10-18T03:42:53.722 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 826 kB/s | 90 kB 00:00 2023-10-18T03:42:53.771 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2023-10-18T03:42:53.775 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:42:53.776 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2023-10-18T03:42:53.776 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:42:53.790 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:42:53.791 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:42:53.901 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:42:53.903 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:42:54.430 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:42:54.618 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-18T03:42:54.673 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-18T03:42:54.729 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-18T03:42:54.765 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-18T03:42:54.807 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-18T03:42:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-18T03:42:55.013 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 7/13 2023-10-18T03:42:55.067 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-18T03:42:55.830 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-18T03:42:55.880 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-18T03:42:55.914 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-18T03:42:55.960 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-18T03:42:55.983 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:18.0.0-6794.gfbf795c8.el9.noarch 13/13 2023-10-18T03:42:56.817 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6794.gfbf795c8.el9.noarch 13/13 2023-10-18T03:42:56.818 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6794.gfbf795c8.el9.noarch 1/13 2023-10-18T03:42:56.818 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-18T03:42:56.818 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-18T03:42:56.818 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-18T03:42:56.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-18T03:42:56.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-18T03:42:56.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-18T03:42:56.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-18T03:42:56.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-18T03:42:56.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 10/13 2023-10-18T03:42:56.820 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-18T03:42:56.820 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-18T03:42:57.206 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-18T03:42:57.207 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:57.207 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:42:57.207 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:42:57.207 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-18T03:42:57.208 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-18T03:42:57.208 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-18T03:42:57.208 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-18T03:42:57.208 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-18T03:42:57.208 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-18T03:42:57.209 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-18T03:42:57.209 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-18T03:42:57.209 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-18T03:42:57.209 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-18T03:42:57.209 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-18T03:42:57.210 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-18T03:42:57.210 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:57.210 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:42:57.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2023-10-18T03:42:58.045 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:21 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:42:58.190 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:42:58.191 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-18T03:42:58.191 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:42:58.191 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-18T03:42:58.191 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:42:58.192 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 132 k 2023-10-18T03:42:58.192 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:42:58.192 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-18T03:42:58.192 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-18T03:42:58.192 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-18T03:42:58.193 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-18T03:42:58.193 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-18T03:42:58.193 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:58.193 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:42:58.193 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2023-10-18T03:42:58.194 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-10-18T03:42:58.194 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:42:58.195 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.8 M 2023-10-18T03:42:58.195 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 33 M 2023-10-18T03:42:58.195 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:42:58.506 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.1 MB/s | 228 kB 00:00 2023-10-18T03:42:58.598 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): ceph-mgr-cephadm-18.0.0-6794.gfbf795c8.e 457 kB/s | 132 kB 00:00 2023-10-18T03:42:58.748 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2023-10-18T03:42:58.773 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 333 kB/s | 58 kB 00:00 2023-10-18T03:42:58.882 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2023-10-18T03:42:58.907 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2023-10-18T03:42:58.909 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:42:58.909 INFO:teuthology.orchestra.run.smithi003.stdout:Total 9.4 MB/s | 6.8 MB 00:00 2023-10-18T03:42:58.909 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:42:58.920 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:42:58.920 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:42:59.014 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:42:59.015 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:42:59.312 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:42:59.501 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-18T03:42:59.552 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-18T03:43:00.117 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-18T03:43:00.198 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-18T03:43:00.266 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-18T03:43:00.298 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 6/6 2023-10-18T03:43:01.093 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 6/6 2023-10-18T03:43:01.094 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 1/6 2023-10-18T03:43:01.094 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-18T03:43:01.094 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-18T03:43:01.094 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-18T03:43:01.094 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-18T03:43:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-18T03:43:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:01.413 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:43:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:43:01.413 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-18T03:43:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-18T03:43:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-18T03:43:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-18T03:43:01.414 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-18T03:43:01.415 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:01.415 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:01.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2023-10-18T03:43:02.254 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:25 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:02.398 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:02.399 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:43:02.399 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:02.400 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:43:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 831 k 2023-10-18T03:43:02.400 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-10-18T03:43:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-18T03:43:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:02.401 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:43:02.401 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:02.401 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2023-10-18T03:43:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:02.402 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 909 k 2023-10-18T03:43:02.402 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 3.0 M 2023-10-18T03:43:02.402 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:43:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 633 kB/s | 79 kB 00:00 2023-10-18T03:43:02.842 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-18.0.0-6794.gfbf795c8.el9.x86_ 1.8 MB/s | 831 kB 00:00 2023-10-18T03:43:02.843 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:43:02.843 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.0 MB/s | 909 kB 00:00 2023-10-18T03:43:02.843 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:43:02.865 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:43:02.866 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:43:02.945 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:43:02.945 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:43:03.141 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:43:03.287 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-18T03:43:03.361 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:18.0.0-6794.gfbf795c8.el9.x86_64 2/2 2023-10-18T03:43:04.074 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6794.gfbf795c8.el9.x86_64 2/2 2023-10-18T03:43:04.075 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/2 2023-10-18T03:43:04.385 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-18T03:43:04.385 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:04.385 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:43:04.386 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:18.0.0-6794.gfbf795c8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-18T03:43:04.386 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:04.386 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:04.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-volume 2023-10-18T03:43:05.250 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:28 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:05.394 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:05.395 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:05.395 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-10-18T03:43:05.395 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:05.396 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:43:05.396 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:18.0.0-6794.gfbf795c8.el9 ceph-noarch 262 k 2023-10-18T03:43:05.396 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:05.396 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:43:05.396 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:05.397 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-18T03:43:05.397 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:05.397 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 262 k 2023-10-18T03:43:05.397 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.2 M 2023-10-18T03:43:05.397 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:43:05.621 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-volume-18.0.0-6794.gfbf795c8.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-10-18T03:43:05.621 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:43:05.622 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-10-18T03:43:05.622 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:43:05.630 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:43:05.630 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:43:05.670 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:43:05.670 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:43:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:43:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:18.0.0-6794.gfbf795c8.el9.noarch 1/1 2023-10-18T03:43:05.980 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:18.0.0-6794.gfbf795c8.el9.noarch 1/1 2023-10-18T03:43:05.981 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-18T03:43:05.981 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-18T03:43:05.981 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:07.027 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:18.0.0-6794.gfbf795c8.el9.noarch 1/1 2023-10-18T03:43:07.028 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:07.028 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:43:07.028 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:18.0.0-6794.gfbf795c8.el9.noarch 2023-10-18T03:43:07.029 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:07.029 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:07.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2023-10-18T03:43:07.952 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:30 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:08.098 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:08.098 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:08.098 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-18T03:43:08.099 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:08.099 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:43:08.099 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 127 k 2023-10-18T03:43:08.099 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:08.100 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:43:08.100 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:08.100 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-18T03:43:08.100 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:08.100 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 127 k 2023-10-18T03:43:08.101 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 456 k 2023-10-18T03:43:08.101 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:43:08.533 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-18.0.0-6794.gfbf795c8.el9.x86_64 290 kB/s | 127 kB 00:00 2023-10-18T03:43:08.534 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:43:08.534 INFO:teuthology.orchestra.run.smithi003.stdout:Total 290 kB/s | 127 kB 00:00 2023-10-18T03:43:08.535 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:43:08.540 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:43:08.540 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:43:08.600 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:43:08.600 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:43:08.748 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:43:08.842 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:09.287 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:09.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:09.593 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:09.593 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:43:09.593 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:43:09.593 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:09.593 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:09.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2023-10-18T03:43:10.454 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:33 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:10.547 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:43:10.599 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:10.601 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:43:10.601 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:10.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2023-10-18T03:43:11.285 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:34 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:11.430 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:11.431 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-18T03:43:11.431 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:11.431 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:43:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 31 k 2023-10-18T03:43:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:11.432 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:43:11.432 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:11.432 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-18T03:43:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:11.433 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 31 k 2023-10-18T03:43:11.433 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 139 k 2023-10-18T03:43:11.433 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:43:11.595 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-18.0.0-6794.gfbf795c8.el9.x86_6 191 kB/s | 31 kB 00:00 2023-10-18T03:43:11.596 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:43:11.596 INFO:teuthology.orchestra.run.smithi003.stdout:Total 189 kB/s | 31 kB 00:00 2023-10-18T03:43:11.596 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:43:11.599 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:43:11.599 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:43:11.627 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:43:11.628 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:43:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:43:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:12.535 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:12.536 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:12.536 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:43:12.536 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:43:12.537 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:12.537 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:12.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2023-10-18T03:43:13.318 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:36 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:13.410 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:43:13.463 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:13.463 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:43:13.464 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:13.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2023-10-18T03:43:14.143 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:37 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:14.235 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:43:14.287 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:14.289 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:43:14.289 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:14.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2023-10-18T03:43:14.966 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:37 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:15.060 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:43:15.113 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:15.113 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:43:15.114 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:15.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2023-10-18T03:43:15.796 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:38 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:15.890 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:43:15.942 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:15.943 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:43:15.943 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:16.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2023-10-18T03:43:16.636 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:39 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:16.727 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:43:16.779 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:16.780 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:43:16.781 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:16.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2023-10-18T03:43:17.464 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:40 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:17.556 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:18.0.0-6794.gfbf795c8.el9.x86_64 is already installed. 2023-10-18T03:43:17.611 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:17.612 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-10-18T03:43:17.612 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:17.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2023-10-18T03:43:18.297 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:41 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:18.444 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:18.445 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:18.445 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-10-18T03:43:18.445 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:18.446 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:43:18.446 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 86 k 2023-10-18T03:43:18.446 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:18.446 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:43:18.446 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:18.447 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-18T03:43:18.447 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:18.447 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 86 k 2023-10-18T03:43:18.447 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 252 k 2023-10-18T03:43:18.448 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:43:18.648 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-18.0.0-6794.gfbf795c8.el9.x86_64.rpm 428 kB/s | 86 kB 00:00 2023-10-18T03:43:18.648 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:43:18.649 INFO:teuthology.orchestra.run.smithi003.stdout:Total 424 kB/s | 86 kB 00:00 2023-10-18T03:43:18.649 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:43:18.656 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:43:18.657 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:43:18.715 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:43:18.715 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:43:18.857 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:43:18.959 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:19.401 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:19.765 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:19.765 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:19.765 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:43:19.765 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:43:19.766 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:19.766 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:19.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2023-10-18T03:43:20.593 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:43 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:20.740 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:20.741 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:20.741 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-10-18T03:43:20.742 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:20.742 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:43:20.742 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 3.0 M 2023-10-18T03:43:20.742 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:20.742 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:43:20.743 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:20.743 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-18T03:43:20.743 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:20.743 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.0 M 2023-10-18T03:43:20.743 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 13 M 2023-10-18T03:43:20.744 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:43:21.434 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-18.0.0-6794.gfbf795c8.el9.x86_64.rpm 4.3 MB/s | 3.0 MB 00:00 2023-10-18T03:43:21.435 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:43:21.435 INFO:teuthology.orchestra.run.smithi003.stdout:Total 4.3 MB/s | 3.0 MB 00:00 2023-10-18T03:43:21.436 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:43:21.446 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:43:21.446 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:43:21.524 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:43:21.524 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:43:21.972 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:43:22.043 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:22.069 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:22.070 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-18T03:43:22.070 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-18T03:43:22.070 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-18T03:43:22.070 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-18T03:43:22.071 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:23.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:23.241 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:23.242 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:43:23.242 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:43:23.242 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:23.243 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:23.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2023-10-18T03:43:24.095 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:47 ago on Wed 18 Oct 2023 03:40:37 AM UTC. 2023-10-18T03:43:24.241 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-10-18T03:43:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:24.242 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-10-18T03:43:24.243 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:24.243 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-10-18T03:43:24.243 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:18.0.0-6794.gfbf795c8.el9 ceph 168 k 2023-10-18T03:43:24.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:24.244 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-10-18T03:43:24.244 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-10-18T03:43:24.244 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-10-18T03:43:24.244 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:24.244 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 168 k 2023-10-18T03:43:24.244 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 532 k 2023-10-18T03:43:24.245 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-10-18T03:43:24.454 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-18.0.0-6794.gfbf795c8.el9.x86_64.rpm 800 kB/s | 168 kB 00:00 2023-10-18T03:43:24.455 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-10-18T03:43:24.455 INFO:teuthology.orchestra.run.smithi003.stdout:Total 795 kB/s | 168 kB 00:00 2023-10-18T03:43:24.455 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-10-18T03:43:24.463 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-10-18T03:43:24.463 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-10-18T03:43:24.523 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-10-18T03:43:24.524 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-10-18T03:43:24.681 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-10-18T03:43:24.788 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:25.233 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:25.548 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:18.0.0-6794.gfbf795c8.el9.x86_64 1/1 2023-10-18T03:43:25.548 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:25.549 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-10-18T03:43:25.549 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:18.0.0-6794.gfbf795c8.el9.x86_64 2023-10-18T03:43:25.549 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:25.549 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-10-18T03:43:25.772 DEBUG:teuthology.parallel:result is None 2023-10-18T03:43:25.773 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=fbf795c854382dc4ab29c95372096a3ba925ed95 2023-10-18T03:43:26.020 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-18T03:43:26.042 INFO:teuthology.orchestra.run.smithi003.stdout:18.0.0-6794.gfbf795c8.el9 2023-10-18T03:43:26.043 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6794.gfbf795c8.el9 2023-10-18T03:43:26.043 INFO:teuthology.task.install:The correct ceph version 18.0.0-6794.gfbf795c8 is installed. 2023-10-18T03:43:26.044 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-18T03:43:26.044 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:26.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-18T03:43:26.113 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-18T03:43:26.114 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:26.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2023-10-18T03:43:26.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-18T03:43:26.255 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-18T03:43:26.255 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:26.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-18T03:43:26.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-18T03:43:26.402 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-18T03:43:26.402 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:26.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2023-10-18T03:43:26.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-18T03:43:26.545 INFO:teuthology.run_tasks:Running task ceph... 2023-10-18T03:43:26.627 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-18T03:43:26.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2023-10-18T03:43:26.658 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-18T03:43:26.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-18T03:43:26.729 INFO:tasks.ceph:Creating extra log directories... 2023-10-18T03:43:26.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-18T03:43:26.807 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-18T03:43:26.808 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-18T03:43:26.808 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264', 'branch': 'wip-crimson-scrub-testing-2023-10-17', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7431264', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-crimson-scrub-testing-2023-10-17'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95'}}, 'workunit': {'branch': 'wip-crimson-scrub-testing-2023-10-17', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95'}}, 'owner': 'scheduled_sjust@teuthology', 'priority': 51, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 4400, 'sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-crimson-scrub-testing-2023-10-17', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95', 'targets': {'smithi003.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5XHdjHt2RFShvGSuGm1VMFOoHPjtSMZgUHDtXvvx4lESLwrSniIqeicsTbOvTtgUbPojTArnahW1WOutSZ4tw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(POOL_FULL\\)'], 'sha1': 'fbf795c854382dc4ab29c95372096a3ba925ed95', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}, 'env': {'RBD_FEATURES': '61'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-18_03:24:35', 'tube': 'smithi', 'user': 'sjust', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-18T03:43:26.809 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-18T03:43:26.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2023-10-18T03:43:26.926 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:26.926 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2023-10-18T03:43:26.982 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-18T03:43:26.982 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2023-10-18T03:43:27.037 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-18T03:43:27.037 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T03:43:27.038 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-10-18T03:43:27.038 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T03:43:27.038 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T03:43:27.038 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-10-18 03:43:22.322174203 +0000 2023-10-18T03:43:27.038 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-10-18 03:39:52.100975565 +0000 2023-10-18T03:43:27.039 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-10-18 03:39:52.100975565 +0000 2023-10-18T03:43:27.039 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-10-18 03:39:52.100975565 +0000 2023-10-18T03:43:27.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-18T03:43:27.106 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-10-18T03:43:27.106 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-10-18T03:43:27.107 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000194469 s, 2.6 MB/s 2023-10-18T03:43:27.108 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-18T03:43:27.163 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2023-10-18T03:43:27.218 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-18T03:43:27.219 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T03:43:27.219 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-10-18T03:43:27.219 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T03:43:27.219 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T03:43:27.219 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-10-18 03:43:22.322174203 +0000 2023-10-18T03:43:27.220 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-10-18 03:39:52.608968991 +0000 2023-10-18T03:43:27.220 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-10-18 03:39:52.608968991 +0000 2023-10-18T03:43:27.220 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-10-18 03:39:52.608968991 +0000 2023-10-18T03:43:27.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-18T03:43:27.287 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-10-18T03:43:27.288 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-10-18T03:43:27.288 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000209838 s, 2.4 MB/s 2023-10-18T03:43:27.289 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-18T03:43:27.345 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2023-10-18T03:43:27.400 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-18T03:43:27.401 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T03:43:27.401 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-10-18T03:43:27.401 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T03:43:27.401 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T03:43:27.401 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-10-18 03:43:22.322174203 +0000 2023-10-18T03:43:27.402 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-10-18 03:39:53.108962519 +0000 2023-10-18T03:43:27.402 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-10-18 03:39:53.108962519 +0000 2023-10-18T03:43:27.402 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-10-18 03:39:53.108962519 +0000 2023-10-18T03:43:27.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-18T03:43:27.469 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-10-18T03:43:27.469 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-10-18T03:43:27.469 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000204989 s, 2.5 MB/s 2023-10-18T03:43:27.470 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-18T03:43:27.526 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2023-10-18T03:43:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-18T03:43:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-18T03:43:27.582 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-10-18T03:43:27.582 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-18T03:43:27.582 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2023-10-18T03:43:27.582 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2023-10-18 03:43:22.323174190 +0000 2023-10-18T03:43:27.583 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2023-10-18 03:39:53.602956126 +0000 2023-10-18T03:43:27.583 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2023-10-18 03:39:53.602956126 +0000 2023-10-18T03:43:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2023-10-18 03:39:53.602956126 +0000 2023-10-18T03:43:27.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-18T03:43:27.650 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2023-10-18T03:43:27.650 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2023-10-18T03:43:27.651 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000210607 s, 2.4 MB/s 2023-10-18T03:43:27.653 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-18T03:43:27.709 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T03:43:27.710 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-18T03:43:27.710 INFO:tasks.ceph:Generating config... 2023-10-18T03:43:27.711 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-10-18T03:43:27.711 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-10-18T03:43:27.712 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-10-18T03:43:27.712 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-10-18T03:43:27.712 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-10-18T03:43:27.713 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-18T03:43:27.713 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-18T03:43:27.713 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-18T03:43:27.713 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-18T03:43:27.714 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-18T03:43:27.714 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-18T03:43:27.714 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-18T03:43:27.715 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-18T03:43:27.715 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-18T03:43:27.715 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-18T03:43:27.716 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-18T03:43:27.716 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-18T03:43:27.716 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-18T03:43:27.717 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-18T03:43:27.717 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-18T03:43:27.717 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-18T03:43:27.717 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-18T03:43:27.718 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-18T03:43:27.718 INFO:tasks.ceph:Setting up mon.a... 2023-10-18T03:43:27.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-18T03:43:27.811 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2023-10-18T03:43:27.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-18T03:43:27.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-18T03:43:27.899 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.3')] 2023-10-18T03:43:27.899 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.3', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-18T03:43:27.900 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:27.901 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-18T03:43:27.954 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.3 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-18T03:43:28.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:28.038+0000 7f884ca2ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:28.040 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:28.038+0000 7f884ca2ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:28.041 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:28.038+0000 7f884ca2ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:28.042 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-18T03:43:28.042 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid 1df30998-f818-4d6d-87b8-b14ebc272db7 2023-10-18T03:43:28.042 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = pacific 2023-10-18T03:43:28.042 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2023-10-18T03:43:28.043 INFO:teuthology.orchestra.run.smithi003.stdout:fsid 1df30998-f818-4d6d-87b8-b14ebc272db7 2023-10-18T03:43:28.043 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2023-10-18T03:43:28.038925+0000 2023-10-18T03:43:28.043 INFO:teuthology.orchestra.run.smithi003.stdout:created 2023-10-18T03:43:28.038925+0000 2023-10-18T03:43:28.043 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 16 (pacific) 2023-10-18T03:43:28.043 INFO:teuthology.orchestra.run.smithi003.stdout:election_strategy: 1 2023-10-18T03:43:28.044 INFO:teuthology.orchestra.run.smithi003.stdout:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2023-10-18T03:43:28.044 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-18T03:43:28.044 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-18T03:43:28.059 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1df30998-f818-4d6d-87b8-b14ebc272db7... 2023-10-18T03:43:28.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-18T03:43:28.159 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2023-10-18T03:43:28.160 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2023-10-18T03:43:28.160 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-18T03:43:28.160 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2023-10-18T03:43:28.160 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.161 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2023-10-18T03:43:28.161 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2023-10-18T03:43:28.161 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.161 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2023-10-18T03:43:28.162 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2023-10-18T03:43:28.162 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.162 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2023-10-18T03:43:28.162 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2023-10-18T03:43:28.162 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.163 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-18T03:43:28.163 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2023-10-18T03:43:28.163 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.163 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2023-10-18T03:43:28.163 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-18T03:43:28.163 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.164 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2023-10-18T03:43:28.164 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.164 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2023-10-18T03:43:28.164 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2023-10-18T03:43:28.164 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2023-10-18T03:43:28.165 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2023-10-18T03:43:28.165 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2023-10-18T03:43:28.165 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2023-10-18T03:43:28.165 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2023-10-18T03:43:28.165 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-18T03:43:28.166 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-18T03:43:28.166 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2023-10-18T03:43:28.166 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.166 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-18T03:43:28.166 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.167 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2023-10-18T03:43:28.167 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.167 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2023-10-18T03:43:28.167 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.167 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2023-10-18T03:43:28.168 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2023-10-18T03:43:28.168 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2023-10-18T03:43:28.168 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = 172.21.15.3 2023-10-18T03:43:28.168 INFO:teuthology.orchestra.run.smithi003.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-18T03:43:28.168 INFO:teuthology.orchestra.run.smithi003.stdout: ms client mode = crc 2023-10-18T03:43:28.169 INFO:teuthology.orchestra.run.smithi003.stdout: ms cluster mode = crc 2023-10-18T03:43:28.169 INFO:teuthology.orchestra.run.smithi003.stdout: ms mon client mode = crc 2023-10-18T03:43:28.169 INFO:teuthology.orchestra.run.smithi003.stdout: ms mon cluster mode = crc 2023-10-18T03:43:28.169 INFO:teuthology.orchestra.run.smithi003.stdout: ms mon service mode = crc 2023-10-18T03:43:28.169 INFO:teuthology.orchestra.run.smithi003.stdout: ms service mode = crc 2023-10-18T03:43:28.169 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 1df30998-f818-4d6d-87b8-b14ebc272db7 2023-10-18T03:43:28.170 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.170 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2023-10-18T03:43:28.170 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2023-10-18T03:43:28.170 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.170 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2023-10-18T03:43:28.171 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2023-10-18T03:43:28.171 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2023-10-18T03:43:28.171 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.171 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2023-10-18T03:43:28.171 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2023-10-18T03:43:28.172 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.172 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2023-10-18T03:43:28.172 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2023-10-18T03:43:28.173 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2023-10-18T03:43:28.174 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.174 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2023-10-18T03:43:28.174 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2023-10-18T03:43:28.174 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.174 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2023-10-18T03:43:28.175 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.175 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2023-10-18T03:43:28.175 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.175 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2023-10-18T03:43:28.175 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.176 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2023-10-18T03:43:28.176 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2023-10-18T03:43:28.176 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.176 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2023-10-18T03:43:28.176 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2023-10-18T03:43:28.177 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2023-10-18T03:43:28.177 INFO:teuthology.orchestra.run.smithi003.stdout: crimson seastar smp = 3 2023-10-18T03:43:28.177 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2023-10-18T03:43:28.177 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2023-10-18T03:43:28.177 INFO:teuthology.orchestra.run.smithi003.stdout: osd shutdown pgref assert = True 2023-10-18T03:43:28.178 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.178 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2023-10-18T03:43:28.178 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2023-10-18T03:43:28.182 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2023-10-18T03:43:28.182 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2023-10-18T03:43:28.182 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2023-10-18T03:43:28.182 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2023-10-18T03:43:28.182 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2023-10-18T03:43:28.183 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2023-10-18T03:43:28.183 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.183 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2023-10-18T03:43:28.183 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2023-10-18T03:43:28.184 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2023-10-18T03:43:28.184 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2023-10-18T03:43:28.184 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2023-10-18T03:43:28.184 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2023-10-18T03:43:28.184 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2023-10-18T03:43:28.185 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2023-10-18T03:43:28.185 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2023-10-18T03:43:28.185 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2023-10-18T03:43:28.185 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2023-10-18T03:43:28.185 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.186 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-18T03:43:28.186 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2023-10-18T03:43:28.186 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2023-10-18T03:43:28.186 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.186 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2023-10-18T03:43:28.187 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-18T03:43:28.187 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-18T03:43:28.187 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.187 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2023-10-18T03:43:28.187 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2023-10-18T03:43:28.188 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.188 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2023-10-18T03:43:28.188 INFO:teuthology.orchestra.run.smithi003.stdout: mon min osdmap epochs = 50 2023-10-18T03:43:28.188 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune interval = 2 2023-10-18T03:43:28.188 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune min = 15 2023-10-18T03:43:28.188 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune txsize = 2 2023-10-18T03:43:28.189 INFO:teuthology.orchestra.run.smithi003.stdout: paxos service trim min = 10 2023-10-18T03:43:28.189 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default crimson = True 2023-10-18T03:43:28.189 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:28.189 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2023-10-18T03:43:28.189 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2023-10-18T03:43:28.190 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2023-10-18T03:43:28.190 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2023-10-18T03:43:28.190 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-18T03:43:28.190 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-18T03:43:28.190 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2023-10-18T03:43:28.192 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-18T03:43:28.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-18T03:43:28.257 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-18T03:43:28.258 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:28.258 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-18T03:43:28.273 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:28.273 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-18T03:43:28.328 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T03:43:28.329 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:28.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-18T03:43:28.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-18T03:43:28.415 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:28.416 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-18T03:43:28.470 INFO:tasks.ceph:Setting up mon nodes... 2023-10-18T03:43:28.471 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-18T03:43:28.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-18T03:43:28.578 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-18T03:43:28.581 INFO:tasks.ceph:Setting up mds nodes... 2023-10-18T03:43:28.581 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-18T03:43:28.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-18T03:43:28.633 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-18T03:43:28.652 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-18T03:43:28.652 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-18T03:43:28.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-18T03:43:28.683 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T03:43:28.683 INFO:tasks.ceph:role: osd.0 2023-10-18T03:43:28.684 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T03:43:28.684 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-18T03:43:28.763 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T03:43:28.763 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T03:43:28.763 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T03:43:28.763 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-18T03:43:28.764 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T03:43:28.764 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2023-10-18T03:43:28.764 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T03:43:28.764 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-18T03:43:28.764 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T03:43:28.765 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T03:43:29.263 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2023-10-18T03:43:29.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2023-10-18T03:43:29.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-18T03:43:29.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-18T03:43:29.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-18T03:43:29.518 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T03:43:29.519 INFO:tasks.ceph:role: osd.1 2023-10-18T03:43:29.519 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T03:43:29.520 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-18T03:43:29.599 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T03:43:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T03:43:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T03:43:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-18T03:43:29.600 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T03:43:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2023-10-18T03:43:29.600 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T03:43:29.600 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-18T03:43:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T03:43:29.601 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T03:43:30.040 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2023-10-18T03:43:30.042 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2023-10-18T03:43:30.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-18T03:43:30.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-18T03:43:30.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-18T03:43:30.233 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-18T03:43:30.233 INFO:tasks.ceph:role: osd.2 2023-10-18T03:43:30.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T03:43:30.234 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-18T03:43:30.311 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-18T03:43:30.311 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-18T03:43:30.311 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-18T03:43:30.311 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-18T03:43:30.312 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-18T03:43:30.312 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2023-10-18T03:43:30.312 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-18T03:43:30.312 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-18T03:43:30.312 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-18T03:43:30.312 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-18T03:43:30.794 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2023-10-18T03:43:30.796 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2023-10-18T03:43:30.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-18T03:43:30.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-18T03:43:30.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T03:43:31.516 INFO:teuthology.orchestra.run.smithi003.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-18T03:43:31.517 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:31,516 seastar - Reactor backend: linux-aio 2023-10-18T03:43:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:31,532 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-18T03:43:31.534 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:31,532 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-18T03:43:31.535 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:31,532 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-18T03:43:31.535 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:31,532 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-18T03:43:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:==80535==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-18T03:43:31.542 INFO:teuthology.orchestra.run.smithi003.stderr:==80535==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd50dc1000; bottom 0x7f67b940b000; size: 0x0095979b6000 (642493669376) 2023-10-18T03:43:31.543 INFO:teuthology.orchestra.run.smithi003.stderr:False positive error reports may follow 2023-10-18T03:43:31.543 INFO:teuthology.orchestra.run.smithi003.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-18T03:43:31.587 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:31,585 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-18T03:43:31.587 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:31,585 [shard 0] osd - get_early_config: set --smp 3 2023-10-18T03:43:32.106 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:32.106 INFO:teuthology.orchestra.run.smithi003.stderr:Suppressions used: 2023-10-18T03:43:32.107 INFO:teuthology.orchestra.run.smithi003.stderr: count bytes template 2023-10-18T03:43:32.107 INFO:teuthology.orchestra.run.smithi003.stderr: 1 8 MallocExtension::Register 2023-10-18T03:43:32.107 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:32.107 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-18T03:43:32.158 INFO:teuthology.orchestra.run.smithi003.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-18T03:43:32.158 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:32,157 seastar - Reactor backend: linux-aio 2023-10-18T03:43:32.176 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:32,175 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-18T03:43:32.177 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:32,175 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-18T03:43:32.177 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:32,175 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-18T03:43:32.184 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:32,183 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-18T03:43:32.187 INFO:teuthology.orchestra.run.smithi003.stderr:==80533==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-18T03:43:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:==80533==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd50dc1000; bottom 0x7f67b7204000; size: 0x009599bbd000 (642529349632) 2023-10-18T03:43:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:False positive error reports may follow 2023-10-18T03:43:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-18T03:43:35.819 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:35.819 INFO:teuthology.orchestra.run.smithi003.stderr:Suppressions used: 2023-10-18T03:43:35.820 INFO:teuthology.orchestra.run.smithi003.stderr: count bytes template 2023-10-18T03:43:35.820 INFO:teuthology.orchestra.run.smithi003.stderr: 1 8 MallocExtension::Register 2023-10-18T03:43:35.820 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:35.820 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-18T03:43:35.821 INFO:teuthology.orchestra.run.smithi003.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 1df30998-f818-4d6d-87b8-b14ebc272db7 2023-10-18T03:43:35.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-18T03:43:35.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T03:43:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-18T03:43:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:36,437 seastar - Reactor backend: linux-aio 2023-10-18T03:43:36.454 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:36,452 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-18T03:43:36.454 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:36,452 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-18T03:43:36.454 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:36,452 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-18T03:43:36.454 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:36,452 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-18T03:43:36.456 INFO:teuthology.orchestra.run.smithi003.stderr:==81343==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-18T03:43:36.462 INFO:teuthology.orchestra.run.smithi003.stderr:==81343==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdd88eb000; bottom 0x7faabe0aa000; size: 0x00531a841000 (356927148032) 2023-10-18T03:43:36.462 INFO:teuthology.orchestra.run.smithi003.stderr:False positive error reports may follow 2023-10-18T03:43:36.463 INFO:teuthology.orchestra.run.smithi003.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-18T03:43:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d32586a 0x2d326c18 0x2d328166 0x2d3284ea 0x2d328634 0x2d328a8a 0x54daf 0x2387f60a 0x238ea326 0x237bfd7f 0x237e4f15 0x23691e64 0x236bf8b7 0x25686d71 0x2568c499 0x2568ea34 0x2568fc0d 0x2325df41 0x2329f7f8 0x232c17e7 0x2d2ca94f 0x2d35fd12 0x2d57d2e2 0x2d57f12d 0x2cff498c 0x2cff688d 0x1f6026a6 0x1f60c813 0x1f485d33 0x3feaf 0x3ff5f 0x1ef3fa04 2023-10-18T03:43:36.489 INFO:teuthology.orchestra.run.smithi003.stderr:kernel callstack: 2023-10-18T03:43:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:36,497 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-18T03:43:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:36,497 [shard 0] osd - get_early_config: set --smp 3 2023-10-18T03:43:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:Suppressions used: 2023-10-18T03:43:36.997 INFO:teuthology.orchestra.run.smithi003.stderr: count bytes template 2023-10-18T03:43:36.997 INFO:teuthology.orchestra.run.smithi003.stderr: 1 8 MallocExtension::Register 2023-10-18T03:43:36.997 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:36.997 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-18T03:43:37.046 INFO:teuthology.orchestra.run.smithi003.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-18T03:43:37.046 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:37,045 seastar - Reactor backend: linux-aio 2023-10-18T03:43:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:37,063 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-18T03:43:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:37,063 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-18T03:43:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:37,063 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-18T03:43:37.072 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:37,071 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-18T03:43:37.074 INFO:teuthology.orchestra.run.smithi003.stderr:==81341==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-18T03:43:37.080 INFO:teuthology.orchestra.run.smithi003.stderr:==81341==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdd88eb000; bottom 0x7faabbea3000; size: 0x00531ca48000 (356962828288) 2023-10-18T03:43:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:False positive error reports may follow 2023-10-18T03:43:37.081 INFO:teuthology.orchestra.run.smithi003.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-18T03:43:40.703 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:40.703 INFO:teuthology.orchestra.run.smithi003.stderr:Suppressions used: 2023-10-18T03:43:40.704 INFO:teuthology.orchestra.run.smithi003.stderr: count bytes template 2023-10-18T03:43:40.704 INFO:teuthology.orchestra.run.smithi003.stderr: 1 8 MallocExtension::Register 2023-10-18T03:43:40.704 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:40.704 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-18T03:43:40.705 INFO:teuthology.orchestra.run.smithi003.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 1df30998-f818-4d6d-87b8-b14ebc272db7 2023-10-18T03:43:40.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-18T03:43:40.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-18T03:43:41.339 INFO:teuthology.orchestra.run.smithi003.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-18T03:43:41.340 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,339 seastar - Reactor backend: linux-aio 2023-10-18T03:43:41.360 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,358 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-18T03:43:41.360 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,358 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-18T03:43:41.361 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,358 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-18T03:43:41.361 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,358 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-18T03:43:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:==82151==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-18T03:43:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:==82151==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc7c754000; bottom 0x7fd065e79000; size: 0x002c168db000 (189356945408) 2023-10-18T03:43:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:False positive error reports may follow 2023-10-18T03:43:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-18T03:43:41.394 INFO:teuthology.orchestra.run.smithi003.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d32586a 0x2d326c18 0x2d328166 0x2d3284ea 0x2d328634 0x2d328a8a 0x54daf 0x238eba15 0x237bf528 0x237c0315 0x237e66a9 0x23691e64 0x236bf8b7 0x25686d71 0x2568c499 0x2568ea34 0x2568fc0d 0x2325df41 0x2329f7f8 0x232c17e7 0x2d2ca94f 0x2d35fd12 0x2d57d2e2 0x2d57f12d 0x2cff498c 0x2cff688d 0x1f6026a6 0x1f60c813 0x1f485d33 0x3feaf 0x3ff5f 0x1ef3fa04 2023-10-18T03:43:41.395 INFO:teuthology.orchestra.run.smithi003.stderr:kernel callstack: 2023-10-18T03:43:41.404 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,403 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-18T03:43:41.405 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,403 [shard 0] osd - get_early_config: set --smp 3 2023-10-18T03:43:41.903 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:41.903 INFO:teuthology.orchestra.run.smithi003.stderr:Suppressions used: 2023-10-18T03:43:41.903 INFO:teuthology.orchestra.run.smithi003.stderr: count bytes template 2023-10-18T03:43:41.904 INFO:teuthology.orchestra.run.smithi003.stderr: 1 8 MallocExtension::Register 2023-10-18T03:43:41.904 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:41.904 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-18T03:43:41.959 INFO:teuthology.orchestra.run.smithi003.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-18T03:43:41.967 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,959 seastar - Reactor backend: linux-aio 2023-10-18T03:43:41.978 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,976 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-18T03:43:41.978 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,976 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-18T03:43:41.978 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,976 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-18T03:43:41.985 INFO:teuthology.orchestra.run.smithi003.stderr:INFO 2023-10-18 03:43:41,984 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-18T03:43:41.988 INFO:teuthology.orchestra.run.smithi003.stderr:==82149==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-18T03:43:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:==82149==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc7c754000; bottom 0x7fd063c72000; size: 0x002c18ae2000 (189392625664) 2023-10-18T03:43:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:False positive error reports may follow 2023-10-18T03:43:41.994 INFO:teuthology.orchestra.run.smithi003.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-18T03:43:45.616 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:45.616 INFO:teuthology.orchestra.run.smithi003.stderr:Suppressions used: 2023-10-18T03:43:45.616 INFO:teuthology.orchestra.run.smithi003.stderr: count bytes template 2023-10-18T03:43:45.617 INFO:teuthology.orchestra.run.smithi003.stderr: 1 8 MallocExtension::Register 2023-10-18T03:43:45.617 INFO:teuthology.orchestra.run.smithi003.stderr:----------------------------------------------------- 2023-10-18T03:43:45.617 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-18T03:43:45.617 INFO:teuthology.orchestra.run.smithi003.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 1df30998-f818-4d6d-87b8-b14ebc272db7 2023-10-18T03:43:45.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-18T03:43:45.685 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-18T03:43:45.686 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:45.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-18T03:43:45.756 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:45.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-18T03:43:45.826 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:45.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-18T03:43:45.896 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:45.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-18T03:43:45.966 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:45.967 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-18T03:43:46.021 INFO:tasks.ceph:Adding keys to all mons... 2023-10-18T03:43:46.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-18T03:43:46.090 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2023-10-18T03:43:46.090 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBgVC9luytlIhAAmnHEUBnxHYzBKE1sdunYuQ== 2023-10-18T03:43:46.090 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2023-10-18T03:43:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBkVC9ltbZhDhAA/jCu2zaT8GQCv+rB3PLqLg== 2023-10-18T03:43:46.091 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2023-10-18T03:43:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBpVC9lTQG1BxAAfBvcmRGPgn53JmX6JeffdQ== 2023-10-18T03:43:46.091 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2023-10-18T03:43:46.091 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBuVC9ltU2DAhAAMA3u+V2+E8d/WeLXRqcw9A== 2023-10-18T03:43:46.092 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2023-10-18T03:43:46.092 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBgVC9lcCmtJRAAEZBkGBxQmNHRUiETfHhF5g== 2023-10-18T03:43:46.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-18T03:43:46.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T03:43:46.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T03:43:46.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-18T03:43:46.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-18T03:43:46.448 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-18T03:43:46.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-18T03:43:46.480 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-18T03:43:46.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:46.581+0000 7fb01220d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:46.584 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:46.581+0000 7fb01220d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:46.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-18T03:43:46.630 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-18T03:43:46.685 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-18T03:43:46.686 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-18T03:43:46.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-18T03:43:46.727 INFO:tasks.ceph.mon.a:Started 2023-10-18T03:43:46.728 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-18T03:43:46.728 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-18T03:43:46.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-18T03:43:46.731 INFO:tasks.ceph.mgr.x:Started 2023-10-18T03:43:46.731 INFO:tasks.ceph:Setting crush tunables to default 2023-10-18T03:43:46.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2023-10-18T03:43:46.814 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-18T03:43:46.811+0000 7f6a8104f980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:46.814 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-18T03:43:46.811+0000 7f6a8104f980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:46.814 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-18T03:43:46.811+0000 7f6a8104f980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:46.815 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:46.811+0000 7f9a02c1cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:46.846 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:46.838+0000 7f28a482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:46.873 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:46.871+0000 7f28a482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:46.945 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2023-10-18T03:43:46.956 INFO:tasks.ceph:check_enable_crimson: True 2023-10-18T03:43:46.956 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-18T03:43:46.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-18T03:43:47.019 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:47.016+0000 7f9a02c1cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:47.020 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:47.017+0000 7f9a02c1cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:47.066 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:47.059+0000 7f53dd11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:47.069 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:47.067+0000 7f53dd11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:47.151 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-18T03:43:47.149+0000 7f6a78ff1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-18T03:43:47.152 INFO:tasks.ceph.mon.a.smithi003.stderr:Please be aware that this feature is experimental, untested, 2023-10-18T03:43:47.152 INFO:tasks.ceph.mon.a.smithi003.stderr:unsupported, and may result in data corruption, data loss, 2023-10-18T03:43:47.152 INFO:tasks.ceph.mon.a.smithi003.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-18T03:43:47.152 INFO:tasks.ceph.mon.a.smithi003.stderr:feature with important data. 2023-10-18T03:43:47.152 INFO:tasks.ceph.mon.a.smithi003.stderr: 2023-10-18T03:43:47.481 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:47.478+0000 7f9a02c1cf00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-18T03:43:47.535 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:47.532+0000 7f9a02c1cf00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-18T03:43:47.957 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-18T03:43:47.958 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:47.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-18T03:43:47.989 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:47.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-18T03:43:48.060 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:43:48.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-18T03:43:48.087 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.085+0000 7f9a02c1cf00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-18T03:43:48.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new 1c5e1f9f-9de0-4c66-811b-734162151957 0 2023-10-18T03:43:48.143 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.141+0000 7f9a02c1cf00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-18T03:43:48.191 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.189+0000 7f9a02c1cf00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-18T03:43:48.250 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.247+0000 7f9a02c1cf00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-18T03:43:48.254 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:48.246+0000 7fab8979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:48.257 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:48.255+0000 7fab8979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:48.310 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.308+0000 7f9a02c1cf00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-18T03:43:48.356 INFO:teuthology.orchestra.run.smithi003.stdout:0 2023-10-18T03:43:48.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new d148002c-6b96-4254-909b-13c4cab6c3a6 1 2023-10-18T03:43:48.418 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.416+0000 7f9a02c1cf00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-18T03:43:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:48.469+0000 7f34ab8f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:48.477+0000 7f34ab8f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:48.495 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.493+0000 7f9a02c1cf00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-18T03:43:48.544 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.541+0000 7f9a02c1cf00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-18T03:43:48.559 INFO:teuthology.orchestra.run.smithi003.stdout:1 2023-10-18T03:43:48.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd new 57b695f7-5f05-4cd7-9683-cc6431c067b9 2 2023-10-18T03:43:48.600 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.597+0000 7f9a02c1cf00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-18T03:43:48.651 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:48.649+0000 7f9a02c1cf00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-18T03:43:48.681 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:48.673+0000 7f65f7617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:48.683 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:48.681+0000 7f65f7617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:48.763 INFO:teuthology.orchestra.run.smithi003.stdout:2 2023-10-18T03:43:48.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-18T03:43:48.850 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-18T03:43:48.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-18T03:43:48.893 INFO:tasks.ceph.osd.0:Started 2023-10-18T03:43:48.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-18T03:43:48.929 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-18T03:43:48.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-18T03:43:48.972 INFO:tasks.ceph.osd.1:Started 2023-10-18T03:43:48.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-18T03:43:49.010 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-18T03:43:49.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-18T03:43:49.053 INFO:tasks.ceph.osd.2:Started 2023-10-18T03:43:49.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T03:43:49.072 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:49.070+0000 7f9a02c1cf00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-18T03:43:49.241 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:49.238+0000 7f9a02c1cf00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-18T03:43:49.278 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:49.268+0000 7ff97f79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:49.283 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:49.281+0000 7ff97f79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:49.405 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:49.402+0000 7f9a02c1cf00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-18T03:43:49.435 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:49.436 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":6,"fsid":"1df30998-f818-4d6d-87b8-b14ebc272db7","created":"2023-10-18T03:43:46.860473+0000","modified":"2023-10-18T03:43:48.761298+0000","last_up_change":"0.000000","last_in_change":"2023-10-18T03:43:48.761298+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1c5e1f9f-9de0-4c66-811b-734162151957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d148002c-6b96-4254-909b-13c4cab6c3a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"57b695f7-5f05-4cd7-9683-cc6431c067b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T03:43:49.455 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-18T03:43:49.456 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-18T03:43:49.459 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:49.456+0000 7f9a02c1cf00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-18T03:43:49.571 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:49.568+0000 7f9a02c1cf00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-18T03:43:49.674 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:49.671+0000 7f9a02c1cf00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-18T03:43:49.758 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-18T03:43:49.830 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:49.827+0000 7f9a02c1cf00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-18T03:43:49.860 INFO:teuthology.misc.health.smithi003.stderr:2023-10-18T03:43:49.854+0000 7fe84ef04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:49.864 INFO:teuthology.misc.health.smithi003.stderr:2023-10-18T03:43:49.861+0000 7fe84ef04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:49.877 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:49.875+0000 7f9a02c1cf00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-18T03:43:49.931 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-18T03:43:49.932 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":6,"fsid":"1df30998-f818-4d6d-87b8-b14ebc272db7","created":"2023-10-18T03:43:46.860473+0000","modified":"2023-10-18T03:43:48.761298+0000","last_up_change":"0.000000","last_in_change":"2023-10-18T03:43:48.761298+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1c5e1f9f-9de0-4c66-811b-734162151957","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d148002c-6b96-4254-909b-13c4cab6c3a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"57b695f7-5f05-4cd7-9683-cc6431c067b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T03:43:49.942 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-18T03:43:50.034 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:50.032+0000 7f9a02c1cf00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-18T03:43:50.898 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:50.895+0000 7f9a02c1cf00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-18T03:43:51.196 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:51.193+0000 7f9a02c1cf00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-18T03:43:51.254 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:51.251+0000 7f9a02c1cf00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-18T03:43:51.374 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:51.371+0000 7f9a02c1cf00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-18T03:43:52.395 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T03:43:52.392+0000 7f99a2d15640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-18T03:43:56.244 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-18T03:43:56.355 INFO:teuthology.misc.health.smithi003.stderr:2023-10-18T03:43:56.349+0000 7f0bca908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:56.358 INFO:teuthology.misc.health.smithi003.stderr:2023-10-18T03:43:56.356+0000 7f0bca908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:56.633 INFO:teuthology.misc.health.smithi003.stdout: 2023-10-18T03:43:56.633 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"1df30998-f818-4d6d-87b8-b14ebc272db7","created":"2023-10-18T03:43:46.860473+0000","modified":"2023-10-18T03:43:56.460405+0000","last_up_change":"2023-10-18T03:43:53.447564+0000","last_in_change":"2023-10-18T03:43:48.761298+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-18T03:43:55.447281+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c5e1f9f-9de0-4c66-811b-734162151957","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":3716380902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3716380902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3716380902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3716380902}]},"public_addr":"172.21.15.3:6803/3716380902","cluster_addr":"172.21.15.3:6802/3716380902","heartbeat_back_addr":"172.21.15.3:6809/3716380902","heartbeat_front_addr":"172.21.15.3:6808/3716380902","state":["exists","up"]},{"osd":1,"uuid":"d148002c-6b96-4254-909b-13c4cab6c3a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":3954762116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3954762116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3954762116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3954762116}]},"public_addr":"172.21.15.3:6805/3954762116","cluster_addr":"172.21.15.3:6804/3954762116","heartbeat_back_addr":"172.21.15.3:6811/3954762116","heartbeat_front_addr":"172.21.15.3:6810/3954762116","state":["exists","up"]},{"osd":2,"uuid":"57b695f7-5f05-4cd7-9683-cc6431c067b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":365423771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":365423771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":365423771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":365423771}]},"public_addr":"172.21.15.3:6807/365423771","cluster_addr":"172.21.15.3:6806/365423771","heartbeat_back_addr":"172.21.15.3:6812/365423771","heartbeat_front_addr":"172.21.15.3:6813/365423771","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T03:43:56.645 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-18T03:43:56.646 INFO:tasks.ceph:Creating RBD pool 2023-10-18T03:43:56.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-18T03:43:56.763 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:56.754+0000 7f5182f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:56.767 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:56.764+0000 7f5182f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:57.468 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'rbd' created 2023-10-18T03:43:57.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-18T03:43:57.613 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:57.603+0000 7fd61c51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:57.616 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:57.614+0000 7fd61c51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:58.473 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-18T03:43:58.491 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-18T03:43:58.492 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-18T03:43:58.492 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-18T03:43:58.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-18T03:43:58.493 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-18T03:43:58.632 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:58.626+0000 7fa1a399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:58.635 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:58.633+0000 7fa1a399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:58.870 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:58.880 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3029459608},{"type":"v1","addr":"172.21.15.3:6801","nonce":3029459608}]},"active_addr":"172.21.15.3:6801/3029459608","active_change":"2023-10-18T03:43:51.386066+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":771424190}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2948375870}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2577383448}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":207991659}]}]} 2023-10-18T03:43:58.883 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-18T03:43:58.883 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-18T03:43:58.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T03:43:58.995 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:58.989+0000 7f228875e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:58.998 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:58.995+0000 7f228875e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.184 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:59.185 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"fsid":"1df30998-f818-4d6d-87b8-b14ebc272db7","created":"2023-10-18T03:43:46.860473+0000","modified":"2023-10-18T03:43:58.467490+0000","last_up_change":"2023-10-18T03:43:53.447564+0000","last_in_change":"2023-10-18T03:43:48.761298+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-18T03:43:55.447281+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T03:43:56.990771+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c5e1f9f-9de0-4c66-811b-734162151957","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":3716380902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3716380902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3716380902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3716380902}]},"public_addr":"172.21.15.3:6803/3716380902","cluster_addr":"172.21.15.3:6802/3716380902","heartbeat_back_addr":"172.21.15.3:6809/3716380902","heartbeat_front_addr":"172.21.15.3:6808/3716380902","state":["exists","up"]},{"osd":1,"uuid":"d148002c-6b96-4254-909b-13c4cab6c3a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":3954762116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3954762116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3954762116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3954762116}]},"public_addr":"172.21.15.3:6805/3954762116","cluster_addr":"172.21.15.3:6804/3954762116","heartbeat_back_addr":"172.21.15.3:6811/3954762116","heartbeat_front_addr":"172.21.15.3:6810/3954762116","state":["exists","up"]},{"osd":2,"uuid":"57b695f7-5f05-4cd7-9683-cc6431c067b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":365423771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":365423771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":365423771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":365423771}]},"public_addr":"172.21.15.3:6807/365423771","cluster_addr":"172.21.15.3:6806/365423771","heartbeat_back_addr":"172.21.15.3:6812/365423771","heartbeat_front_addr":"172.21.15.3:6813/365423771","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T03:43:59.196 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-18T03:43:59.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-18T03:43:59.309 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.303+0000 7fed4c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.312 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.309+0000 7fed4c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.505 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:43:59.506 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":13,"fsid":"1df30998-f818-4d6d-87b8-b14ebc272db7","created":"2023-10-18T03:43:46.860473+0000","modified":"2023-10-18T03:43:59.471831+0000","last_up_change":"2023-10-18T03:43:53.447564+0000","last_in_change":"2023-10-18T03:43:48.761298+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-18T03:43:55.447281+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-18T03:43:56.990771+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1c5e1f9f-9de0-4c66-811b-734162151957","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":3716380902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3716380902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3716380902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3716380902}]},"public_addr":"172.21.15.3:6803/3716380902","cluster_addr":"172.21.15.3:6802/3716380902","heartbeat_back_addr":"172.21.15.3:6809/3716380902","heartbeat_front_addr":"172.21.15.3:6808/3716380902","state":["exists","up"]},{"osd":1,"uuid":"d148002c-6b96-4254-909b-13c4cab6c3a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":3954762116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3954762116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3954762116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3954762116}]},"public_addr":"172.21.15.3:6805/3954762116","cluster_addr":"172.21.15.3:6804/3954762116","heartbeat_back_addr":"172.21.15.3:6811/3954762116","heartbeat_front_addr":"172.21.15.3:6810/3954762116","state":["exists","up"]},{"osd":2,"uuid":"57b695f7-5f05-4cd7-9683-cc6431c067b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":365423771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":365423771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":365423771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":365423771}]},"public_addr":"172.21.15.3:6807/365423771","cluster_addr":"172.21.15.3:6806/365423771","heartbeat_back_addr":"172.21.15.3:6812/365423771","heartbeat_front_addr":"172.21.15.3:6813/365423771","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-18T03:43:59.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-18T03:43:59.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-18T03:43:59.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-18T03:43:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.631+0000 7f7170e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.638 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.631+0000 7f277279e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.639 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.631+0000 7f9b1c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.641 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.639+0000 7f7170e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.639+0000 7f277279e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.642 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.639+0000 7f9b1c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.674 INFO:teuthology.orchestra.run.smithi003.stdout:34359738374 2023-10-18T03:43:59.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-18T03:43:59.676 INFO:teuthology.orchestra.run.smithi003.stdout:34359738374 2023-10-18T03:43:59.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-18T03:43:59.677 INFO:teuthology.orchestra.run.smithi003.stdout:34359738374 2023-10-18T03:43:59.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-18T03:43:59.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.826+0000 7ff29902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.833 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.826+0000 7f42a799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.834 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.826+0000 7f6232a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.836 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.834+0000 7ff29902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.834+0000 7f42a799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:43:59.837 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:43:59.834+0000 7f6232a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:44:00.024 INFO:teuthology.orchestra.run.smithi003.stdout:34359738374 2023-10-18T03:44:00.028 INFO:teuthology.orchestra.run.smithi003.stdout:34359738374 2023-10-18T03:44:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:34359738374 2023-10-18T03:44:00.038 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-18T03:44:00.038 DEBUG:teuthology.parallel:result is None 2023-10-18T03:44:00.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2023-10-18T03:44:00.043 DEBUG:teuthology.parallel:result is None 2023-10-18T03:44:00.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2023-10-18T03:44:00.047 DEBUG:teuthology.parallel:result is None 2023-10-18T03:44:00.047 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-18T03:44:00.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-18T03:44:00.199 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:44:00.193+0000 7fd4a5d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:44:00.202 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:44:00.199+0000 7fd4a5d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:44:00.384 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:44:00.385 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-18T03:44:00.396 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-18T03:43:59.661101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82548,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490252,"statfs":{"total":322122547200,"available":322038018048,"internally_reserved":0,"allocated":1347584,"data_stored":1230629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.589906+0000","last_change":"2023-10-18T03:43:58.590319+0000","last_active":"2023-10-18T03:43:58.589906+0000","last_peered":"2023-10-18T03:43:58.589906+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.586170+0000","last_became_peered":"2023-10-18T03:43:58.586170+0000","last_unstale":"2023-10-18T03:43:58.589906+0000","last_undegraded":"2023-10-18T03:43:58.589906+0000","last_fullsized":"2023-10-18T03:43:58.589906+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:59.484405+0000","last_change":"2023-10-18T03:43:58.614807+0000","last_active":"2023-10-18T03:43:59.484405+0000","last_peered":"2023-10-18T03:43:59.484405+0000","last_clean":"2023-10-18T03:43:59.484405+0000","last_became_active":"2023-10-18T03:43:58.611468+0000","last_became_peered":"2023-10-18T03:43:58.611468+0000","last_unstale":"2023-10-18T03:43:59.484405+0000","last_undegraded":"2023-10-18T03:43:59.484405+0000","last_fullsized":"2023-10-18T03:43:59.484405+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:59.484655+0000","last_change":"2023-10-18T03:43:58.593671+0000","last_active":"2023-10-18T03:43:59.484655+0000","last_peered":"2023-10-18T03:43:59.484655+0000","last_clean":"2023-10-18T03:43:59.484655+0000","last_became_active":"2023-10-18T03:43:58.590457+0000","last_became_peered":"2023-10-18T03:43:58.590457+0000","last_unstale":"2023-10-18T03:43:59.484655+0000","last_undegraded":"2023-10-18T03:43:59.484655+0000","last_fullsized":"2023-10-18T03:43:59.484655+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.580422+0000","last_change":"2023-10-18T03:43:58.580860+0000","last_active":"2023-10-18T03:43:58.580422+0000","last_peered":"2023-10-18T03:43:58.580422+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.569171+0000","last_became_peered":"2023-10-18T03:43:58.569171+0000","last_unstale":"2023-10-18T03:43:58.580422+0000","last_undegraded":"2023-10-18T03:43:58.580422+0000","last_fullsized":"2023-10-18T03:43:58.580422+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.577410+0000","last_change":"2023-10-18T03:43:58.577766+0000","last_active":"2023-10-18T03:43:58.577410+0000","last_peered":"2023-10-18T03:43:58.577410+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.574526+0000","last_became_peered":"2023-10-18T03:43:58.574526+0000","last_unstale":"2023-10-18T03:43:58.577410+0000","last_undegraded":"2023-10-18T03:43:58.577410+0000","last_fullsized":"2023-10-18T03:43:58.577410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.601650+0000","last_change":"2023-10-18T03:43:58.602048+0000","last_active":"2023-10-18T03:43:58.601650+0000","last_peered":"2023-10-18T03:43:58.601650+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.598935+0000","last_became_peered":"2023-10-18T03:43:58.598935+0000","last_unstale":"2023-10-18T03:43:58.601650+0000","last_undegraded":"2023-10-18T03:43:58.601650+0000","last_fullsized":"2023-10-18T03:43:58.601650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.594496+0000","last_change":"2023-10-18T03:43:58.594989+0000","last_active":"2023-10-18T03:43:58.594496+0000","last_peered":"2023-10-18T03:43:58.594496+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.591243+0000","last_became_peered":"2023-10-18T03:43:58.591243+0000","last_unstale":"2023-10-18T03:43:58.594496+0000","last_undegraded":"2023-10-18T03:43:58.594496+0000","last_fullsized":"2023-10-18T03:43:58.594496+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:59.483171+0000","last_change":"2023-10-18T03:43:58.588407+0000","last_active":"2023-10-18T03:43:59.483171+0000","last_peered":"2023-10-18T03:43:59.483171+0000","last_clean":"2023-10-18T03:43:59.483171+0000","last_became_active":"2023-10-18T03:43:58.584973+0000","last_became_peered":"2023-10-18T03:43:58.584973+0000","last_unstale":"2023-10-18T03:43:59.483171+0000","last_undegraded":"2023-10-18T03:43:59.483171+0000","last_fullsized":"2023-10-18T03:43:59.483171+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.508096+0000","last_change":"2023-10-18T03:43:56.505970+0000","last_active":"2023-10-18T03:43:58.508096+0000","last_peered":"2023-10-18T03:43:58.508096+0000","last_clean":"2023-10-18T03:43:58.508096+0000","last_became_active":"2023-10-18T03:43:56.500807+0000","last_became_peered":"2023-10-18T03:43:56.500807+0000","last_unstale":"2023-10-18T03:43:58.508096+0000","last_undegraded":"2023-10-18T03:43:58.508096+0000","last_fullsized":"2023-10-18T03:43:58.508096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:55.454075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:55.454075+0000","last_clean_scrub_stamp":"2023-10-18T03:43:55.454075+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27004,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830596,"statfs":{"total":107374182400,"available":107346530304,"internally_reserved":0,"allocated":53248,"data_stored":16631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-18T03:44:00.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-18T03:44:00.508 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:44:00.502+0000 7f70efe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:44:00.511 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:44:00.508+0000 7f70efe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:44:00.693 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:44:00.693 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2023-10-18T03:44:00.704 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-18T03:43:59.661101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82548,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490252,"statfs":{"total":322122547200,"available":322038018048,"internally_reserved":0,"allocated":1347584,"data_stored":1230629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.589906+0000","last_change":"2023-10-18T03:43:58.590319+0000","last_active":"2023-10-18T03:43:58.589906+0000","last_peered":"2023-10-18T03:43:58.589906+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.586170+0000","last_became_peered":"2023-10-18T03:43:58.586170+0000","last_unstale":"2023-10-18T03:43:58.589906+0000","last_undegraded":"2023-10-18T03:43:58.589906+0000","last_fullsized":"2023-10-18T03:43:58.589906+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:59.484405+0000","last_change":"2023-10-18T03:43:58.614807+0000","last_active":"2023-10-18T03:43:59.484405+0000","last_peered":"2023-10-18T03:43:59.484405+0000","last_clean":"2023-10-18T03:43:59.484405+0000","last_became_active":"2023-10-18T03:43:58.611468+0000","last_became_peered":"2023-10-18T03:43:58.611468+0000","last_unstale":"2023-10-18T03:43:59.484405+0000","last_undegraded":"2023-10-18T03:43:59.484405+0000","last_fullsized":"2023-10-18T03:43:59.484405+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:59.484655+0000","last_change":"2023-10-18T03:43:58.593671+0000","last_active":"2023-10-18T03:43:59.484655+0000","last_peered":"2023-10-18T03:43:59.484655+0000","last_clean":"2023-10-18T03:43:59.484655+0000","last_became_active":"2023-10-18T03:43:58.590457+0000","last_became_peered":"2023-10-18T03:43:58.590457+0000","last_unstale":"2023-10-18T03:43:59.484655+0000","last_undegraded":"2023-10-18T03:43:59.484655+0000","last_fullsized":"2023-10-18T03:43:59.484655+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.580422+0000","last_change":"2023-10-18T03:43:58.580860+0000","last_active":"2023-10-18T03:43:58.580422+0000","last_peered":"2023-10-18T03:43:58.580422+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.569171+0000","last_became_peered":"2023-10-18T03:43:58.569171+0000","last_unstale":"2023-10-18T03:43:58.580422+0000","last_undegraded":"2023-10-18T03:43:58.580422+0000","last_fullsized":"2023-10-18T03:43:58.580422+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.577410+0000","last_change":"2023-10-18T03:43:58.577766+0000","last_active":"2023-10-18T03:43:58.577410+0000","last_peered":"2023-10-18T03:43:58.577410+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.574526+0000","last_became_peered":"2023-10-18T03:43:58.574526+0000","last_unstale":"2023-10-18T03:43:58.577410+0000","last_undegraded":"2023-10-18T03:43:58.577410+0000","last_fullsized":"2023-10-18T03:43:58.577410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.601650+0000","last_change":"2023-10-18T03:43:58.602048+0000","last_active":"2023-10-18T03:43:58.601650+0000","last_peered":"2023-10-18T03:43:58.601650+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.598935+0000","last_became_peered":"2023-10-18T03:43:58.598935+0000","last_unstale":"2023-10-18T03:43:58.601650+0000","last_undegraded":"2023-10-18T03:43:58.601650+0000","last_fullsized":"2023-10-18T03:43:58.601650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.594496+0000","last_change":"2023-10-18T03:43:58.594989+0000","last_active":"2023-10-18T03:43:58.594496+0000","last_peered":"2023-10-18T03:43:58.594496+0000","last_clean":"2023-10-18T03:43:57.463850+0000","last_became_active":"2023-10-18T03:43:58.591243+0000","last_became_peered":"2023-10-18T03:43:58.591243+0000","last_unstale":"2023-10-18T03:43:58.594496+0000","last_undegraded":"2023-10-18T03:43:58.594496+0000","last_fullsized":"2023-10-18T03:43:58.594496+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:59.483171+0000","last_change":"2023-10-18T03:43:58.588407+0000","last_active":"2023-10-18T03:43:59.483171+0000","last_peered":"2023-10-18T03:43:59.483171+0000","last_clean":"2023-10-18T03:43:59.483171+0000","last_became_active":"2023-10-18T03:43:58.584973+0000","last_became_peered":"2023-10-18T03:43:58.584973+0000","last_unstale":"2023-10-18T03:43:59.483171+0000","last_undegraded":"2023-10-18T03:43:59.483171+0000","last_fullsized":"2023-10-18T03:43:59.483171+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:57.463850+0000","last_clean_scrub_stamp":"2023-10-18T03:43:57.463850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-18T03:43:58.508096+0000","last_change":"2023-10-18T03:43:56.505970+0000","last_active":"2023-10-18T03:43:58.508096+0000","last_peered":"2023-10-18T03:43:58.508096+0000","last_clean":"2023-10-18T03:43:58.508096+0000","last_became_active":"2023-10-18T03:43:56.500807+0000","last_became_peered":"2023-10-18T03:43:56.500807+0000","last_unstale":"2023-10-18T03:43:58.508096+0000","last_undegraded":"2023-10-18T03:43:58.508096+0000","last_fullsized":"2023-10-18T03:43:58.508096+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-18T03:43:55.454075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-18T03:43:55.454075+0000","last_clean_scrub_stamp":"2023-10-18T03:43:55.454075+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27004,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830596,"statfs":{"total":107374182400,"available":107346530304,"internally_reserved":0,"allocated":53248,"data_stored":16631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-18T03:44:00.705 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-18T03:44:00.705 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-18T03:44:00.706 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-18T03:44:00.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-18T03:44:00.816 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:44:00.811+0000 7fcf5e91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:44:00.818 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T03:44:00.816+0000 7fcf5e91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T03:44:01.038 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-10-18T03:44:01.039 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-18T03:44:01.050 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-18T03:44:01.051 INFO:teuthology.run_tasks:Running task workunit... 2023-10-18T03:44:01.063 INFO:tasks.workunit:Pulling workunits from ref fbf795c854382dc4ab29c95372096a3ba925ed95 2023-10-18T03:44:01.064 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-18T03:44:01.064 DEBUG:teuthology.orchestra.run.smithi003:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-18T03:44:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T03:44:01.080 INFO:teuthology.orchestra.run.smithi003.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-18T03:44:01.080 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-18T03:44:01.135 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-18T03:44:01.135 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-18T03:44:01.190 INFO:tasks.workunit:timeout=3h 2023-10-18T03:44:01.191 INFO:tasks.workunit:cleanup=True 2023-10-18T03:44:01.191 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout fbf795c854382dc4ab29c95372096a3ba925ed95 2023-10-18T03:44:01.246 INFO:tasks.workunit.client.0.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-18T03:44:45.238 INFO:tasks.workunit.client.0.smithi003.stderr:Note: switching to 'fbf795c854382dc4ab29c95372096a3ba925ed95'. 2023-10-18T03:44:45.238 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-18T03:44:45.238 INFO:tasks.workunit.client.0.smithi003.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-18T03:44:45.239 INFO:tasks.workunit.client.0.smithi003.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-18T03:44:45.239 INFO:tasks.workunit.client.0.smithi003.stderr:state without impacting any branches by switching back to a branch. 2023-10-18T03:44:45.239 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-18T03:44:45.239 INFO:tasks.workunit.client.0.smithi003.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-18T03:44:45.239 INFO:tasks.workunit.client.0.smithi003.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-18T03:44:45.240 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-18T03:44:45.240 INFO:tasks.workunit.client.0.smithi003.stderr: git switch -c 2023-10-18T03:44:45.240 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-18T03:44:45.240 INFO:tasks.workunit.client.0.smithi003.stderr:Or undo this operation with: 2023-10-18T03:44:45.240 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-18T03:44:45.241 INFO:tasks.workunit.client.0.smithi003.stderr: git switch - 2023-10-18T03:44:45.241 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-18T03:44:45.241 INFO:tasks.workunit.client.0.smithi003.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-18T03:44:45.241 INFO:tasks.workunit.client.0.smithi003.stderr: 2023-10-18T03:44:45.241 INFO:tasks.workunit.client.0.smithi003.stderr:HEAD is now at fbf795c8543 Merge branches sjust/wip-crimson-scrub 2023-10-18T03:44:45.243 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-18T03:44:45.310 INFO:tasks.workunit.client.0.smithi003.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-18T03:44:45.312 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-18T03:44:45.312 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-18T03:44:45.943 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-18T03:44:46.007 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-18T03:44:46.048 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-18T03:44:46.049 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-18T03:44:46.050 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-18T03:44:46.090 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-18T03:44:46.093 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-10-18T03:44:46.093 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-18T03:44:46.149 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2023-10-18T03:44:46.150 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2023-10-18T03:44:46.150 DEBUG:teuthology.orchestra.run.smithi003:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf795c854382dc4ab29c95372096a3ba925ed95 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-10-18T03:44:46.244 INFO:tasks.workunit.client.0.smithi003.stdout:seed 85898 2023-10-18T03:44:46.245 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = TestLibRBD.* 2023-10-18T03:44:46.245 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 169 tests from 1 test suite. 2023-10-18T03:44:46.245 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T03:44:46.245 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 169 tests from TestLibRBD 2023-10-18T03:44:46.282 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.CreateAndStat 2023-10-18T03:44:47.875 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.086 INFO:tasks.workunit.client.0.smithi003.stdout:image has size 2097152 and order 22 2023-10-18T03:44:50.087 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.CreateAndStat (3804 ms) 2023-10-18T03:44:50.087 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2023-10-18T03:44:50.087 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.179 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (92 ms) 2023-10-18T03:44:50.179 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2023-10-18T03:44:50.180 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.279 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.CreateAndStatPP (101 ms) 2023-10-18T03:44:50.280 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.GetId 2023-10-18T03:44:50.280 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.421 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.GetId (142 ms) 2023-10-18T03:44:50.422 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.GetIdPP 2023-10-18T03:44:50.422 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.566 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.GetIdPP (145 ms) 2023-10-18T03:44:50.567 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2023-10-18T03:44:50.567 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.658 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (92 ms) 2023-10-18T03:44:50.658 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2023-10-18T03:44:50.658 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.757 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (98 ms) 2023-10-18T03:44:50.757 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2023-10-18T03:44:50.757 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.847 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (91 ms) 2023-10-18T03:44:50.847 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2023-10-18T03:44:50.848 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:50.940 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (92 ms) 2023-10-18T03:44:50.940 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.OpenAio 2023-10-18T03:44:50.940 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:51.030 INFO:tasks.workunit.client.0.smithi003.stdout:image has size 2097152 and order 22 2023-10-18T03:44:51.030 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.OpenAio (90 ms) 2023-10-18T03:44:51.030 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.OpenAioFail 2023-10-18T03:44:51.037 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.OpenAioFail (7 ms) 2023-10-18T03:44:51.037 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.OpenAioPP 2023-10-18T03:44:51.037 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:51.182 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.OpenAioPP (145 ms) 2023-10-18T03:44:51.182 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2023-10-18T03:44:51.188 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.OpenAioFailPP (7 ms) 2023-10-18T03:44:51.189 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2023-10-18T03:44:51.189 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:51.443 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ResizeAndStat (255 ms) 2023-10-18T03:44:51.443 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2023-10-18T03:44:51.444 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:51.644 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (201 ms) 2023-10-18T03:44:51.644 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2023-10-18T03:44:51.644 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:51.888 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (244 ms) 2023-10-18T03:44:51.888 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2023-10-18T03:44:51.889 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:52.132 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (243 ms) 2023-10-18T03:44:52.132 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2023-10-18T03:44:54.275 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:56.152 INFO:tasks.workunit.client.0.smithi003.stdout:image: image18 2023-10-18T03:44:56.153 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image18 2023-10-18T03:44:56.153 INFO:tasks.workunit.client.0.smithi003.stdout:found image18 2023-10-18T03:44:56.153 INFO:tasks.workunit.client.0.smithi003.stdout:erased image18 2023-10-18T03:44:56.153 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:44:56.654 INFO:tasks.workunit.client.0.smithi003.stdout:image: image18 2023-10-18T03:44:56.654 INFO:tasks.workunit.client.0.smithi003.stdout:image: image19 2023-10-18T03:44:56.654 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image18 2023-10-18T03:44:56.655 INFO:tasks.workunit.client.0.smithi003.stdout:found image18 2023-10-18T03:44:56.655 INFO:tasks.workunit.client.0.smithi003.stdout:erased image18 2023-10-18T03:44:56.655 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image19 2023-10-18T03:44:56.655 INFO:tasks.workunit.client.0.smithi003.stdout:found image19 2023-10-18T03:44:56.655 INFO:tasks.workunit.client.0.smithi003.stdout:erased image19 2023-10-18T03:44:56.656 INFO:tasks.workunit.client.0.smithi003.stdout:image: image19 2023-10-18T03:44:56.656 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image19 2023-10-18T03:44:56.656 INFO:tasks.workunit.client.0.smithi003.stdout:found image19 2023-10-18T03:44:56.656 INFO:tasks.workunit.client.0.smithi003.stdout:erased image19 2023-10-18T03:44:56.656 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (4522 ms) 2023-10-18T03:44:56.657 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2023-10-18T03:44:57.923 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 1 2023-10-18T03:45:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:image: image20 2023-10-18T03:45:00.192 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image20 2023-10-18T03:45:00.192 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:00.238 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 2 2023-10-18T03:45:00.239 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:45:00.239 INFO:tasks.workunit.client.0.smithi003.stdout:image: image20 2023-10-18T03:45:00.239 INFO:tasks.workunit.client.0.smithi003.stdout:image: image21 2023-10-18T03:45:00.239 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image20 2023-10-18T03:45:00.240 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image21 2023-10-18T03:45:00.616 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 1 2023-10-18T03:45:00.617 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:00.617 INFO:tasks.workunit.client.0.smithi003.stdout:image: image21 2023-10-18T03:45:00.617 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image21 2023-10-18T03:45:00.617 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (3963 ms) 2023-10-18T03:45:00.617 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCopy 2023-10-18T03:45:01.962 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:07.249 INFO:tasks.workunit.client.0.smithi003.stdout:image: image22 2023-10-18T03:45:07.249 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image22 2023-10-18T03:45:07.250 INFO:tasks.workunit.client.0.smithi003.stdout:found image22 2023-10-18T03:45:07.250 INFO:tasks.workunit.client.0.smithi003.stdout:erased image22 2023-10-18T03:45:07.250 INFO:tasks.workunit.client.0.smithi003.stdout:image: image22 2023-10-18T03:45:07.250 INFO:tasks.workunit.client.0.smithi003.stdout:image: image23 2023-10-18T03:45:07.250 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image22 2023-10-18T03:45:07.251 INFO:tasks.workunit.client.0.smithi003.stdout:found image22 2023-10-18T03:45:07.251 INFO:tasks.workunit.client.0.smithi003.stdout:erased image22 2023-10-18T03:45:07.251 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image23 2023-10-18T03:45:07.251 INFO:tasks.workunit.client.0.smithi003.stdout:found image23 2023-10-18T03:45:07.251 INFO:tasks.workunit.client.0.smithi003.stdout:erased image23 2023-10-18T03:45:07.252 INFO:tasks.workunit.client.0.smithi003.stdout:100.00% done 2023-10-18T03:45:07.252 INFO:tasks.workunit.client.0.smithi003.stdout:image: image22 2023-10-18T03:45:07.252 INFO:tasks.workunit.client.0.smithi003.stdout:image: image23 2023-10-18T03:45:07.252 INFO:tasks.workunit.client.0.smithi003.stdout:image: image24 2023-10-18T03:45:07.252 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image22 2023-10-18T03:45:07.253 INFO:tasks.workunit.client.0.smithi003.stdout:found image22 2023-10-18T03:45:07.253 INFO:tasks.workunit.client.0.smithi003.stdout:erased image22 2023-10-18T03:45:07.253 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image23 2023-10-18T03:45:07.253 INFO:tasks.workunit.client.0.smithi003.stdout:found image23 2023-10-18T03:45:07.254 INFO:tasks.workunit.client.0.smithi003.stdout:erased image23 2023-10-18T03:45:07.254 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image24 2023-10-18T03:45:07.254 INFO:tasks.workunit.client.0.smithi003.stdout:found image24 2023-10-18T03:45:07.254 INFO:tasks.workunit.client.0.smithi003.stdout:erased image24 2023-10-18T03:45:07.254 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCopy (6632 ms) 2023-10-18T03:45:07.255 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCopyPP 2023-10-18T03:45:08.987 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:13.385 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 1 2023-10-18T03:45:13.386 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:13.386 INFO:tasks.workunit.client.0.smithi003.stdout:image: image25 2023-10-18T03:45:13.386 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image25 2023-10-18T03:45:13.501 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 2 2023-10-18T03:45:13.502 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:45:13.502 INFO:tasks.workunit.client.0.smithi003.stdout:image: image25 2023-10-18T03:45:13.502 INFO:tasks.workunit.client.0.smithi003.stdout:image: image26 2023-10-18T03:45:13.503 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image25 2023-10-18T03:45:13.503 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image26 2023-10-18T03:45:13.848 INFO:tasks.workunit.client.0.smithi003.stdout:100.00% done 2023-10-18T03:45:13.848 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 3 2023-10-18T03:45:13.849 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 3 2023-10-18T03:45:13.849 INFO:tasks.workunit.client.0.smithi003.stdout:image: image25 2023-10-18T03:45:13.849 INFO:tasks.workunit.client.0.smithi003.stdout:image: image26 2023-10-18T03:45:13.849 INFO:tasks.workunit.client.0.smithi003.stdout:image: image27 2023-10-18T03:45:13.849 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image25 2023-10-18T03:45:13.850 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image26 2023-10-18T03:45:13.850 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image27 2023-10-18T03:45:14.113 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCopyPP (6865 ms) 2023-10-18T03:45:14.113 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2023-10-18T03:45:16.006 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:26.997 INFO:tasks.workunit.client.0.smithi003.stdout:image: image28 2023-10-18T03:45:26.998 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image28 2023-10-18T03:45:26.998 INFO:tasks.workunit.client.0.smithi003.stdout:found image28 2023-10-18T03:45:26.998 INFO:tasks.workunit.client.0.smithi003.stdout:erased image28 2023-10-18T03:45:26.998 INFO:tasks.workunit.client.0.smithi003.stdout:image: image28 2023-10-18T03:45:26.999 INFO:tasks.workunit.client.0.smithi003.stdout:image: image29 2023-10-18T03:45:26.999 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image28 2023-10-18T03:45:26.999 INFO:tasks.workunit.client.0.smithi003.stdout:found image28 2023-10-18T03:45:26.999 INFO:tasks.workunit.client.0.smithi003.stdout:erased image28 2023-10-18T03:45:26.999 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image29 2023-10-18T03:45:27.000 INFO:tasks.workunit.client.0.smithi003.stdout:found image29 2023-10-18T03:45:27.000 INFO:tasks.workunit.client.0.smithi003.stdout:erased image29 2023-10-18T03:45:27.000 INFO:tasks.workunit.client.0.smithi003.stdout:100.00% done 2023-10-18T03:45:27.000 INFO:tasks.workunit.client.0.smithi003.stdout:image: image28 2023-10-18T03:45:27.000 INFO:tasks.workunit.client.0.smithi003.stdout:image: image29 2023-10-18T03:45:27.000 INFO:tasks.workunit.client.0.smithi003.stdout:image: image30 2023-10-18T03:45:27.001 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image28 2023-10-18T03:45:27.001 INFO:tasks.workunit.client.0.smithi003.stdout:found image28 2023-10-18T03:45:27.001 INFO:tasks.workunit.client.0.smithi003.stdout:erased image28 2023-10-18T03:45:27.001 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image29 2023-10-18T03:45:27.001 INFO:tasks.workunit.client.0.smithi003.stdout:found image29 2023-10-18T03:45:27.002 INFO:tasks.workunit.client.0.smithi003.stdout:erased image29 2023-10-18T03:45:27.002 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image30 2023-10-18T03:45:27.002 INFO:tasks.workunit.client.0.smithi003.stdout:found image30 2023-10-18T03:45:27.002 INFO:tasks.workunit.client.0.smithi003.stdout:erased image30 2023-10-18T03:45:27.002 INFO:tasks.workunit.client.0.smithi003.stdout:image: image28 2023-10-18T03:45:27.003 INFO:tasks.workunit.client.0.smithi003.stdout:image: image29 2023-10-18T03:45:27.003 INFO:tasks.workunit.client.0.smithi003.stdout:image: image30 2023-10-18T03:45:27.003 INFO:tasks.workunit.client.0.smithi003.stdout:image: image31 2023-10-18T03:45:27.003 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image28 2023-10-18T03:45:27.003 INFO:tasks.workunit.client.0.smithi003.stdout:found image28 2023-10-18T03:45:27.004 INFO:tasks.workunit.client.0.smithi003.stdout:erased image28 2023-10-18T03:45:27.004 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image29 2023-10-18T03:45:27.004 INFO:tasks.workunit.client.0.smithi003.stdout:found image29 2023-10-18T03:45:27.004 INFO:tasks.workunit.client.0.smithi003.stdout:erased image29 2023-10-18T03:45:27.004 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image30 2023-10-18T03:45:27.005 INFO:tasks.workunit.client.0.smithi003.stdout:found image30 2023-10-18T03:45:27.005 INFO:tasks.workunit.client.0.smithi003.stdout:erased image30 2023-10-18T03:45:27.005 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image31 2023-10-18T03:45:27.005 INFO:tasks.workunit.client.0.smithi003.stdout:found image31 2023-10-18T03:45:27.005 INFO:tasks.workunit.client.0.smithi003.stdout:erased image31 2023-10-18T03:45:27.006 INFO:tasks.workunit.client.0.smithi003.stdout:image: image28 2023-10-18T03:45:27.006 INFO:tasks.workunit.client.0.smithi003.stdout:image: image29 2023-10-18T03:45:27.006 INFO:tasks.workunit.client.0.smithi003.stdout:image: image30 2023-10-18T03:45:27.006 INFO:tasks.workunit.client.0.smithi003.stdout:image: image31 2023-10-18T03:45:27.006 INFO:tasks.workunit.client.0.smithi003.stdout:image: image32 2023-10-18T03:45:27.006 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image28 2023-10-18T03:45:27.007 INFO:tasks.workunit.client.0.smithi003.stdout:found image28 2023-10-18T03:45:27.007 INFO:tasks.workunit.client.0.smithi003.stdout:erased image28 2023-10-18T03:45:27.007 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image29 2023-10-18T03:45:27.007 INFO:tasks.workunit.client.0.smithi003.stdout:found image29 2023-10-18T03:45:27.007 INFO:tasks.workunit.client.0.smithi003.stdout:erased image29 2023-10-18T03:45:27.008 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image30 2023-10-18T03:45:27.008 INFO:tasks.workunit.client.0.smithi003.stdout:found image30 2023-10-18T03:45:27.008 INFO:tasks.workunit.client.0.smithi003.stdout:erased image30 2023-10-18T03:45:27.008 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image31 2023-10-18T03:45:27.008 INFO:tasks.workunit.client.0.smithi003.stdout:found image31 2023-10-18T03:45:27.009 INFO:tasks.workunit.client.0.smithi003.stdout:erased image31 2023-10-18T03:45:27.009 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image32 2023-10-18T03:45:27.009 INFO:tasks.workunit.client.0.smithi003.stdout:found image32 2023-10-18T03:45:27.009 INFO:tasks.workunit.client.0.smithi003.stdout:erased image32 2023-10-18T03:45:27.009 INFO:tasks.workunit.client.0.smithi003.stdout:100.00% done 2023-10-18T03:45:27.010 INFO:tasks.workunit.client.0.smithi003.stdout:image: image28 2023-10-18T03:45:27.010 INFO:tasks.workunit.client.0.smithi003.stdout:image: image29 2023-10-18T03:45:27.010 INFO:tasks.workunit.client.0.smithi003.stdout:image: image30 2023-10-18T03:45:27.010 INFO:tasks.workunit.client.0.smithi003.stdout:image: image31 2023-10-18T03:45:27.010 INFO:tasks.workunit.client.0.smithi003.stdout:image: image32 2023-10-18T03:45:27.011 INFO:tasks.workunit.client.0.smithi003.stdout:image: image33 2023-10-18T03:45:27.011 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image28 2023-10-18T03:45:27.011 INFO:tasks.workunit.client.0.smithi003.stdout:found image28 2023-10-18T03:45:27.011 INFO:tasks.workunit.client.0.smithi003.stdout:erased image28 2023-10-18T03:45:27.011 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image29 2023-10-18T03:45:27.012 INFO:tasks.workunit.client.0.smithi003.stdout:found image29 2023-10-18T03:45:27.012 INFO:tasks.workunit.client.0.smithi003.stdout:erased image29 2023-10-18T03:45:27.012 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image30 2023-10-18T03:45:27.012 INFO:tasks.workunit.client.0.smithi003.stdout:found image30 2023-10-18T03:45:27.012 INFO:tasks.workunit.client.0.smithi003.stdout:erased image30 2023-10-18T03:45:27.013 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image31 2023-10-18T03:45:27.013 INFO:tasks.workunit.client.0.smithi003.stdout:found image31 2023-10-18T03:45:27.013 INFO:tasks.workunit.client.0.smithi003.stdout:erased image31 2023-10-18T03:45:27.013 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image32 2023-10-18T03:45:27.013 INFO:tasks.workunit.client.0.smithi003.stdout:found image32 2023-10-18T03:45:27.014 INFO:tasks.workunit.client.0.smithi003.stdout:erased image32 2023-10-18T03:45:27.014 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image33 2023-10-18T03:45:27.014 INFO:tasks.workunit.client.0.smithi003.stdout:found image33 2023-10-18T03:45:27.014 INFO:tasks.workunit.client.0.smithi003.stdout:erased image33 2023-10-18T03:45:27.014 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestDeepCopy (12884 ms) 2023-10-18T03:45:27.015 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2023-10-18T03:45:28.859 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:33.596 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 1 2023-10-18T03:45:33.596 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:33.596 INFO:tasks.workunit.client.0.smithi003.stdout:image: image34 2023-10-18T03:45:33.597 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image34 2023-10-18T03:45:33.759 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 2 2023-10-18T03:45:33.760 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:45:33.760 INFO:tasks.workunit.client.0.smithi003.stdout:image: image34 2023-10-18T03:45:33.760 INFO:tasks.workunit.client.0.smithi003.stdout:image: image35 2023-10-18T03:45:33.760 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image34 2023-10-18T03:45:33.760 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image35 2023-10-18T03:45:34.187 INFO:tasks.workunit.client.0.smithi003.stdout:100.00% done 2023-10-18T03:45:34.188 INFO:tasks.workunit.client.0.smithi003.stdout:num images is: 3 2023-10-18T03:45:34.188 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 3 2023-10-18T03:45:34.188 INFO:tasks.workunit.client.0.smithi003.stdout:image: image34 2023-10-18T03:45:34.188 INFO:tasks.workunit.client.0.smithi003.stdout:image: image35 2023-10-18T03:45:34.189 INFO:tasks.workunit.client.0.smithi003.stdout:image: image36 2023-10-18T03:45:34.189 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image34 2023-10-18T03:45:34.189 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image35 2023-10-18T03:45:34.189 INFO:tasks.workunit.client.0.smithi003.stdout:expected = image36 2023-10-18T03:45:34.672 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (7675 ms) 2023-10-18T03:45:34.673 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2023-10-18T03:45:34.673 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:38.304 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 1 2023-10-18T03:45:38.305 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:38.305 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap1 2023-10-18T03:45:38.305 INFO:tasks.workunit.client.0.smithi003.stdout:found snap1 with size 2097152 2023-10-18T03:45:38.305 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 2 2023-10-18T03:45:38.306 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:45:38.306 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap1 2023-10-18T03:45:38.306 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap2 2023-10-18T03:45:38.306 INFO:tasks.workunit.client.0.smithi003.stdout:found snap1 with size 2097152 2023-10-18T03:45:38.306 INFO:tasks.workunit.client.0.smithi003.stdout:found snap2 with size 4194304 2023-10-18T03:45:38.307 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 1 2023-10-18T03:45:38.307 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:38.307 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap2 2023-10-18T03:45:38.307 INFO:tasks.workunit.client.0.smithi003.stdout:found snap2 with size 4194304 2023-10-18T03:45:38.307 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 0 2023-10-18T03:45:38.308 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:45:38.308 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (3632 ms) 2023-10-18T03:45:38.308 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2023-10-18T03:45:38.308 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:40.319 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (2015 ms) 2023-10-18T03:45:40.320 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2023-10-18T03:45:40.320 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:41.254 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 1 2023-10-18T03:45:41.254 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:41.255 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap1 2023-10-18T03:45:41.255 INFO:tasks.workunit.client.0.smithi003.stdout:found snap1 with size 2097152 2023-10-18T03:45:42.181 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 2 2023-10-18T03:45:42.181 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:45:42.181 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap1 2023-10-18T03:45:42.182 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap2 2023-10-18T03:45:42.182 INFO:tasks.workunit.client.0.smithi003.stdout:found snap1 with size 2097152 2023-10-18T03:45:42.182 INFO:tasks.workunit.client.0.smithi003.stdout:found snap2 with size 4194304 2023-10-18T03:45:43.162 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 1 2023-10-18T03:45:43.162 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:43.162 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap2 2023-10-18T03:45:43.162 INFO:tasks.workunit.client.0.smithi003.stdout:found snap2 with size 4194304 2023-10-18T03:45:44.279 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 0 2023-10-18T03:45:44.280 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:45:44.339 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4019 ms) 2023-10-18T03:45:44.339 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2023-10-18T03:45:44.340 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:50.369 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 0 2023-10-18T03:45:50.370 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:45:50.482 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (6144 ms) 2023-10-18T03:45:50.483 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2023-10-18T03:45:50.483 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:51.292 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 1 2023-10-18T03:45:51.292 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:51.292 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap1 2023-10-18T03:45:51.293 INFO:tasks.workunit.client.0.smithi003.stdout:found snap1 with size 2097152 2023-10-18T03:45:52.365 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 2 2023-10-18T03:45:52.366 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:45:52.366 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap1 2023-10-18T03:45:52.366 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap2 2023-10-18T03:45:52.367 INFO:tasks.workunit.client.0.smithi003.stdout:found snap1 with size 2097152 2023-10-18T03:45:52.367 INFO:tasks.workunit.client.0.smithi003.stdout:found snap2 with size 4194304 2023-10-18T03:45:52.417 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 2 2023-10-18T03:45:52.417 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:45:52.417 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap1-rename 2023-10-18T03:45:52.418 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap2 2023-10-18T03:45:52.418 INFO:tasks.workunit.client.0.smithi003.stdout:found snap1-rename with size 2097152 2023-10-18T03:45:52.418 INFO:tasks.workunit.client.0.smithi003.stdout:found snap2 with size 4194304 2023-10-18T03:45:53.464 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 1 2023-10-18T03:45:53.464 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:45:53.464 INFO:tasks.workunit.client.0.smithi003.stdout:snap: snap2-rename 2023-10-18T03:45:53.464 INFO:tasks.workunit.client.0.smithi003.stdout:found snap2-rename with size 4194304 2023-10-18T03:45:54.327 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 0 2023-10-18T03:45:54.328 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:45:54.379 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (3896 ms) 2023-10-18T03:45:54.379 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2023-10-18T03:45:55.986 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:55.987 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:55.987 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:45:55.987 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:46:00.352 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (5974 ms) 2023-10-18T03:46:00.353 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2023-10-18T03:46:02.549 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:46:05.185 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:46:22.880 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.RemoveFullTry (22528 ms) 2023-10-18T03:46:22.881 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2023-10-18T03:46:27.404 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:46:48.548 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.RemoveFullTryDataPool (25667 ms) 2023-10-18T03:46:48.548 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestIO 2023-10-18T03:46:48.549 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:47:00.157 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:00.158 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:00.158 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:00.158 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:00.158 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:00.158 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.159 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:00.159 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.159 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.159 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:00.159 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.160 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:00.160 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.160 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.160 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:00.160 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.161 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:00.161 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.161 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.161 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:00.161 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.162 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:00.162 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.162 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.162 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:00.162 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.163 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:00.163 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.163 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.163 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:00.163 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:00.163 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:00.164 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:00.164 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:00.164 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:00.164 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:00.164 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:00.165 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:00.165 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:00.165 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:00.165 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.165 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:00.166 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.166 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.166 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:00.166 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.166 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:00.167 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.167 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.167 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:00.167 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.167 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:00.167 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.168 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.168 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:00.168 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.168 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:00.168 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.168 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.169 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:00.169 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.169 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:00.169 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.169 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.169 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:00.170 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.170 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.170 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.170 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.170 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.171 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.171 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:00.171 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:00.171 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:00.171 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.171 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:00.172 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:00.172 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:00.172 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.172 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:00.172 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:00.172 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:00.173 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.173 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:00.173 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:00.173 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:00.173 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.174 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:00.174 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:00.174 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:00.174 INFO:tasks.workunit.client.0.smithi003.stdout:discard: 512~512 = 512 2023-10-18T03:47:00.174 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.175 INFO:tasks.workunit.client.0.smithi003.stdout:aio discard: 1536~512 = 0 2023-10-18T03:47:00.175 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.175 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.175 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.175 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.175 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:00.176 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:47:00.176 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.176 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.176 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:47:00.176 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.177 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.177 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:47:00.177 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.177 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.177 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:47:00.178 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.178 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.178 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.178 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.178 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:47:00.179 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.179 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.179 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:47:00.179 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.179 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.179 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:47:00.180 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.180 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.180 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:47:00.180 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.180 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.181 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.181 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.181 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:47:00.181 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.181 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.182 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:47:00.182 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.182 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.182 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:47:00.182 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.183 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.183 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:47:00.183 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.183 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.183 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.184 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.184 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:47:00.184 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.184 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.184 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:47:00.184 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.185 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.185 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:47:00.185 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.185 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.185 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:47:00.186 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.186 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.186 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.186 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.186 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:47:00.186 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.187 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.187 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:47:00.187 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.187 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.187 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:47:00.188 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.188 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.188 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:47:00.188 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.188 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.188 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.189 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.189 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.189 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:00.189 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.189 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.189 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:00.190 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.190 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.190 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.190 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:00.190 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.191 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:00.192 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.192 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.192 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:00.192 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.192 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.192 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.193 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:00.193 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.193 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.193 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:00.193 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.194 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.194 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.194 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:00.194 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.194 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.195 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:00.195 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:00.195 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.195 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:00.195 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.196 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.196 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:00.196 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.196 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.196 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:00.196 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:00.197 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:00.197 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:00.197 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:00.197 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:00.197 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:00.197 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.007 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.008 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.008 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.008 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.008 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.009 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.009 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.009 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.009 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.009 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.009 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.010 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.010 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.010 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.010 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.010 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:09.011 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.011 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.011 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:09.011 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.011 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.012 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.012 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.012 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.012 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.012 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.013 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.013 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.013 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.013 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.013 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.013 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.014 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.014 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.014 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.014 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.014 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.015 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.015 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.015 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.015 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.015 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.016 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.016 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.016 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.016 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.016 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.016 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.017 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.017 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:09.017 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.017 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.017 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:09.018 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.018 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.018 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.018 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.018 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.019 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.019 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.019 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.019 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.019 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.020 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.020 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.020 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.020 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.020 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.020 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.021 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.021 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.021 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.021 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.021 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.022 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.022 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.022 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.022 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.022 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.023 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.023 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.023 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.023 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.023 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:09.023 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.024 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.024 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:09.024 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.024 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.024 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.025 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.025 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.025 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.025 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.025 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.026 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.026 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.026 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.026 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.026 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.027 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.027 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.027 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.027 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.027 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.028 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.028 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.028 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.028 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.028 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.028 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:09.029 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:09.029 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:09.029 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:09.029 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:09.029 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.030 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.030 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:09.030 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:09.030 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:09.030 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:09.031 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:09.031 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:09.031 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:09.031 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:09.031 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestIO (20459 ms) 2023-10-18T03:47:09.031 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2023-10-18T03:47:09.032 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:47:23.126 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:23.127 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:23.127 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:23.127 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:23.127 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:23.127 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.128 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:23.128 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.128 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.128 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:23.128 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.129 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:23.129 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.129 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.129 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:23.129 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.130 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:23.130 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.130 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.130 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:23.130 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.131 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:23.131 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.131 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.131 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:23.131 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.131 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:23.132 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.132 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.132 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:23.132 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:23.132 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:23.132 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:23.133 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:23.133 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:23.133 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:23.133 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:23.133 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:23.134 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:23.134 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:23.134 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.134 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:23.134 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.135 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.135 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:23.135 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.135 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:23.135 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.136 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.136 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:23.136 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.136 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:23.137 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.137 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.137 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:23.137 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.137 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:23.138 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.138 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.138 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:23.138 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.138 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:23.139 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.139 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.139 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:23.139 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.139 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.140 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.140 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.140 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.140 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.140 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:23.140 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:23.141 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:23.141 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.141 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:23.141 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:23.141 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:23.142 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.142 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:23.142 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:23.142 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:23.142 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.143 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:23.143 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:23.143 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:23.143 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.143 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:23.144 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:23.144 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:23.144 INFO:tasks.workunit.client.0.smithi003.stdout:discard: 512~512 = 512 2023-10-18T03:47:23.144 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.144 INFO:tasks.workunit.client.0.smithi003.stdout:aio discard: 1536~512 = 0 2023-10-18T03:47:23.145 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.145 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.145 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.145 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.145 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:23.146 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:47:23.146 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.146 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.146 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:47:23.147 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.147 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.147 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:47:23.147 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.147 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.147 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:47:23.148 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.148 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.148 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.148 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.148 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:47:23.149 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.149 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.149 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:47:23.149 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.149 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.150 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:47:23.150 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.150 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.150 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:47:23.150 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.151 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.151 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.151 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.151 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:47:23.151 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.152 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.152 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:47:23.152 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.152 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.152 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:47:23.153 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.153 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.153 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:47:23.153 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.153 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.154 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.154 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.154 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:47:23.154 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.154 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.155 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:47:23.155 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.155 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.155 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:47:23.155 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.155 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.156 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:47:23.156 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.156 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.156 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.156 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.157 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:47:23.157 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.157 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.157 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:47:23.157 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.158 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.158 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:47:23.158 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.158 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.158 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:47:23.159 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.159 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.159 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.159 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.159 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.160 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:23.160 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.160 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.160 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:23.160 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.160 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.161 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.161 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:23.161 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.161 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.161 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:23.162 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.162 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.162 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.162 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:23.162 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.163 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.163 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:23.163 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.163 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.163 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.164 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:23.164 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.164 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.164 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:23.164 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.165 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.165 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.165 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:23.165 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.165 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.166 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:23.166 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:23.166 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.166 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:23.166 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.166 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.167 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:23.167 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.167 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.167 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:23.167 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:23.168 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:23.168 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:23.168 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:23.168 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:23.168 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:23.169 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.742 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.742 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.742 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.743 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.743 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.743 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.743 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.743 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.744 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.744 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.744 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.744 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.744 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.745 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.745 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.745 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:36.745 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.745 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.745 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:36.746 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.746 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.746 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.746 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.746 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.747 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.747 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.747 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.747 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.747 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.748 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.748 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.748 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.748 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.749 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.749 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.750 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.750 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.750 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.750 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.750 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.751 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.751 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.751 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.751 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.751 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.752 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.752 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.752 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.752 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.752 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:36.752 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.753 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.753 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:36.753 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.753 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.753 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.754 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.754 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.754 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.754 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.754 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.755 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.755 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.755 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.755 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.755 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.755 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.756 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.756 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.756 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.756 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.756 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.757 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.757 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.757 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.757 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.757 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.757 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.758 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.758 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.758 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.758 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.758 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.759 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:36.759 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.759 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.759 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:36.759 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.760 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.760 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.760 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.760 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.760 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.760 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.761 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.761 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.761 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.761 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.761 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.762 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.762 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.762 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.762 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.762 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.763 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.763 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.763 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.763 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.763 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.764 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.764 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:36.764 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:36.764 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:36.764 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:36.764 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:36.765 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.765 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.765 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:36.765 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:36.765 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:36.766 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:36.766 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:36.766 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:36.766 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:36.766 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:36.767 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 4 2023-10-18T03:47:36.767 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:47:36.767 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:47:36.767 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (27735 ms) 2023-10-18T03:47:36.767 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2023-10-18T03:47:36.768 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:47:54.610 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:54.611 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:54.611 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:54.611 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:54.611 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:47:54.612 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.612 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:54.612 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.612 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.613 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:54.613 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.613 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:54.613 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.613 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.614 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:54.614 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.614 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:54.614 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.614 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.615 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:54.615 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.615 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:54.615 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.615 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.616 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:54.616 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.616 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:47:54.616 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.616 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.617 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:47:54.617 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:54.617 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:54.617 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:54.617 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:54.618 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:54.618 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:54.618 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:54.618 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:54.618 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:47:54.619 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:47:54.619 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.619 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:54.619 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.620 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.620 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:54.620 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.620 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:54.621 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.621 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.621 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:54.621 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.622 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:54.622 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.622 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.622 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:54.622 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.623 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:54.623 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.623 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.623 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:54.623 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.624 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:47:54.624 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.624 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.624 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:47:54.624 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.625 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.625 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.625 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.625 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.625 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.626 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:54.626 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:54.626 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:54.626 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.626 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:54.627 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:54.627 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:54.627 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.627 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:54.628 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:54.628 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:54.628 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.628 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:54.628 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:54.628 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:54.629 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.629 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:47:54.629 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:47:54.629 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:47:54.629 INFO:tasks.workunit.client.0.smithi003.stdout:discard: 512~512 = 512 2023-10-18T03:47:54.630 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.630 INFO:tasks.workunit.client.0.smithi003.stdout:aio discard: 1536~512 = 0 2023-10-18T03:47:54.630 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.630 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.630 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.631 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.631 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:47:54.631 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:47:54.631 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.631 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.632 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:47:54.632 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.632 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.632 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:47:54.632 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.633 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.633 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:47:54.633 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.633 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.633 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.634 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.634 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:47:54.634 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.634 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.635 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:47:54.635 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.635 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.635 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:47:54.635 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.636 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.636 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:47:54.636 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.636 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.636 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.637 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.637 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:47:54.637 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.637 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.638 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:47:54.638 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.638 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.638 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:47:54.638 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.639 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.639 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:47:54.639 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.639 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.639 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.640 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.640 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:47:54.640 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.640 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.640 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:47:54.640 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.641 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.641 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:47:54.641 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.641 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.641 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:47:54.641 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.642 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.642 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.642 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.642 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:47:54.642 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.643 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.643 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:47:54.643 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.643 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.643 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:47:54.644 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.644 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.644 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:47:54.644 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.644 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.645 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.645 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.645 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.645 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:54.645 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.646 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.646 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:54.646 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.646 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.646 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.647 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:54.647 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.647 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.647 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:54.647 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.647 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.648 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.648 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:54.648 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.648 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.648 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:54.649 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.649 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.649 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.649 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:54.649 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.650 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.650 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:54.650 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.650 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.650 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.651 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:54.651 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.651 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.651 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:54.651 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:47:54.652 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.652 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:54.652 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.652 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.652 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:54.653 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.653 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.653 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:47:54.653 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:47:54.653 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:47:54.654 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:47:54.654 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:47:54.654 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:47:54.654 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:47:54.654 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.970 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.970 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.970 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.971 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.971 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.971 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.971 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.971 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.972 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.972 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.972 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.972 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.972 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.973 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.973 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.973 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:48:07.973 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.973 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.974 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:48:07.974 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.974 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.974 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.974 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.975 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.975 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.975 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.975 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.975 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.975 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.976 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.976 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.976 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.976 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.976 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.977 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.977 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.977 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.977 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.977 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.978 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.978 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.978 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.978 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.978 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.979 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.979 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.979 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.979 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.979 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.980 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:48:07.980 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.980 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.980 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:48:07.980 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.981 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.981 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.981 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.981 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.981 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.981 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.982 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.982 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.982 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.982 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.982 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.983 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.983 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.983 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.983 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.983 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.984 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.984 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.984 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.984 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.984 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.984 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.985 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.985 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.985 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.985 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.985 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.986 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.986 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.986 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:48:07.986 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.986 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.987 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:48:07.987 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.987 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.987 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.987 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.988 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.988 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.988 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.988 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.988 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.988 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.989 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.989 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.989 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.989 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.989 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.990 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.990 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.990 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.990 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.990 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.991 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.991 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.991 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.991 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:48:07.991 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:48:07.992 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:48:07.992 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:48:07.992 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:48:07.992 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.992 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.992 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:48:07.993 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:48:07.993 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:48:07.993 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:48:07.993 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:48:07.993 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:48:07.994 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:48:07.994 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:48:07.994 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 4 2023-10-18T03:48:07.994 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:48:07.994 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:48:07.995 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (31228 ms) 2023-10-18T03:48:07.995 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2023-10-18T03:48:07.995 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:49:09.050 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 4 2023-10-18T03:49:09.051 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 4 2023-10-18T03:49:09.051 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 4 2023-10-18T03:49:09.051 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:49:09.052 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:49:09.052 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:49:09.052 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:49:09.052 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:49:09.052 INFO:tasks.workunit.client.0.smithi003.stdout:read: 4 2023-10-18T03:49:09.053 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCloneEncryption (61080 ms) 2023-10-18T03:49:09.053 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2023-10-18T03:49:09.053 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:49:24.762 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (15712 ms) 2023-10-18T03:49:24.762 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2023-10-18T03:49:24.762 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:49:43.271 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (18510 ms) 2023-10-18T03:49:43.272 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2023-10-18T03:49:43.272 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:49:47.842 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4571 ms) 2023-10-18T03:49:47.843 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2023-10-18T03:49:47.843 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:49:53.603 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (5760 ms) 2023-10-18T03:49:53.603 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2023-10-18T03:49:53.603 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_librbd.cc:2838: Skipped 2023-10-18T03:49:53.604 INFO:tasks.workunit.client.0.smithi003.stdout:Skipping due to unmet REQUIRE 2023-10-18T03:49:53.604 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2023-10-18T03:49:53.604 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2023-10-18T03:49:53.604 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:52:12.033 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.EncryptionLoadFormatMismatch (138430 ms) 2023-10-18T03:52:12.034 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EncryptedResize 2023-10-18T03:52:12.034 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:52:28.217 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.EncryptedResize (16182 ms) 2023-10-18T03:52:28.218 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2023-10-18T03:52:28.218 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:52:42.174 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.EncryptedFlattenSmallData (13959 ms) 2023-10-18T03:52:42.174 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2023-10-18T03:52:42.174 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:52:52.747 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:52:52.748 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:52:52.748 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:52:52.748 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:52:52.748 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:52:52.748 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.748 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:52:52.749 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.749 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.749 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:52:52.749 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.749 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:52:52.750 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.750 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.750 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:52:52.750 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.750 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:52:52.751 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.751 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.751 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:52:52.751 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.751 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:52:52.751 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.752 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.752 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:52:52.752 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.752 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:52:52.752 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.753 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.753 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:52:52.753 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:52:52.753 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:52:52.753 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:52:52.753 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:52:52.754 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:52:52.754 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:52:52.754 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:52:52.754 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:52:52.754 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:52:52.755 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:52:52.755 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.755 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:52:52.755 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.755 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.756 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:52:52.756 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.756 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:52:52.756 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.756 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.756 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:52:52.757 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.757 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:52:52.757 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.757 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.757 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:52:52.758 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.758 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:52:52.758 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.758 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.758 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:52:52.759 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.759 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:52:52.759 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.759 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.759 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:52:52.760 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.760 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.760 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.760 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.760 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.761 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.761 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:52:52.761 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:52:52.761 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:52:52.761 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.762 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:52:52.762 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:52:52.762 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:52:52.762 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.762 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:52:52.763 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:52:52.763 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:52:52.763 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.763 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:52:52.763 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:52:52.764 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:52:52.764 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.764 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:52:52.764 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:52:52.764 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:52:52.764 INFO:tasks.workunit.client.0.smithi003.stdout:discard: 512~512 = 512 2023-10-18T03:52:52.765 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.765 INFO:tasks.workunit.client.0.smithi003.stdout:aio discard: 1536~512 = 0 2023-10-18T03:52:52.765 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.765 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.765 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.766 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.766 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:52:52.766 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:52:52.766 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.766 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.766 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:52:52.767 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.767 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.767 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:52:52.767 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.767 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.768 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:52:52.768 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.768 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.768 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.768 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.768 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:52:52.769 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.769 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.769 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:52:52.769 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.769 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.770 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:52:52.770 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.770 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.770 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:52:52.770 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.771 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.771 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.771 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.771 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:52:52.771 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.771 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.772 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:52:52.772 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.772 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.772 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:52:52.772 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.772 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.773 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:52:52.773 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.773 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.773 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.773 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.774 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:52:52.774 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.774 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.774 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:52:52.774 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.775 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.775 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:52:52.775 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.775 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.776 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:52:52.776 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.776 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.776 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.777 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.777 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:52:52.777 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.777 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.778 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:52:52.778 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.778 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.778 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:52:52.778 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.779 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.779 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:52:52.779 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.779 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.780 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.780 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.780 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.780 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:52:52.780 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.781 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.781 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:52:52.781 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.781 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.781 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.782 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:52:52.782 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.782 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.782 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:52:52.782 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.783 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.783 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.783 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:52:52.783 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.783 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.783 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:52:52.784 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.784 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.784 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.784 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:52:52.784 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.785 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.785 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:52:52.785 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.785 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.785 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.786 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:52:52.786 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.786 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.786 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:52:52.786 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:52:52.787 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.787 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:52:52.787 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.787 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.787 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:52:52.787 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.788 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.788 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:52:52.788 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:52:52.788 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:52:52.788 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:52:52.789 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:52:52.789 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:52:52.789 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:52:52.789 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:01.998 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:01.998 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:01.998 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:01.998 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:01.999 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:01.999 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:01.999 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:01.999 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:01.999 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.000 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.000 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.000 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.000 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.000 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.000 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.001 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:02.001 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.001 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.001 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:02.001 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.002 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.002 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.002 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.002 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.002 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.002 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.003 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.003 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.003 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.003 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.003 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.004 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.004 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.004 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.004 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.004 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.004 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.005 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.005 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.005 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.005 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.005 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.006 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.006 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.006 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.006 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.006 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.007 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.007 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.007 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:02.007 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.007 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.008 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:02.008 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.008 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.008 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.008 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.008 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.009 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.009 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.009 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.009 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.009 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.010 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.010 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.010 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.010 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.010 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.011 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.011 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.011 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.011 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.011 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.011 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.012 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.012 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.012 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.012 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.012 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.013 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.013 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.013 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.013 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.013 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:02.014 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.014 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.014 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:02.014 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.014 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.015 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.015 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.015 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.015 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.015 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.015 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.016 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.016 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.016 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.016 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.016 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.017 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.017 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.017 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.017 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.017 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.017 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.018 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.018 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.018 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.018 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.018 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:02.019 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:02.019 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:02.019 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:02.019 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:02.019 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.020 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.020 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:02.020 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:02.020 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:02.020 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:02.020 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:02.021 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:02.021 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:02.021 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (19824 ms) 2023-10-18T03:53:02.021 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2023-10-18T03:53:04.018 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:06.971 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:06.972 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:06.972 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:06.972 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:06.972 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:06.972 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.973 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:06.973 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:06.973 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:06.973 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:06.973 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.974 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:06.974 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:06.974 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:06.974 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:06.974 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.975 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:06.975 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:06.975 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:06.975 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:06.975 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.975 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:06.976 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:06.976 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:06.976 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:06.976 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.977 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:06.977 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:06.977 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:06.977 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:06.977 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.978 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.978 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.978 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.978 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.978 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.979 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:06.979 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:06.979 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:06.979 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.979 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:06.979 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:06.980 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:06.980 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.980 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:06.980 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:06.980 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:06.981 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.981 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:06.981 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:06.981 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:06.981 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:06.981 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:06.982 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:06.982 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:06.982 INFO:tasks.workunit.client.0.smithi003.stdout:discard: 512~512 = 512 2023-10-18T03:53:06.982 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:06.982 INFO:tasks.workunit.client.0.smithi003.stdout:aio discard: 1536~512 = 0 2023-10-18T03:53:06.983 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.983 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.983 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.983 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.983 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:06.983 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:06.984 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:06.984 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestDataPoolIO (4974 ms) 2023-10-18T03:53:06.984 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2023-10-18T03:53:06.984 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:07.139 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (167 ms) 2023-10-18T03:53:07.139 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2023-10-18T03:53:07.139 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:07.305 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (167 ms) 2023-10-18T03:53:07.306 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2023-10-18T03:53:07.306 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:07.503 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (198 ms) 2023-10-18T03:53:07.504 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2023-10-18T03:53:07.504 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:07.747 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (244 ms) 2023-10-18T03:53:07.748 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2023-10-18T03:53:07.748 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:07.990 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (242 ms) 2023-10-18T03:53:07.990 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2023-10-18T03:53:07.990 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:08.223 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (234 ms) 2023-10-18T03:53:08.223 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2023-10-18T03:53:08.223 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:08.470 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (247 ms) 2023-10-18T03:53:08.471 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2023-10-18T03:53:08.471 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:08.704 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (234 ms) 2023-10-18T03:53:08.705 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2023-10-18T03:53:08.705 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:10.013 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (1308 ms) 2023-10-18T03:53:10.013 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2023-10-18T03:53:10.014 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:11.383 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (1371 ms) 2023-10-18T03:53:11.384 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2023-10-18T03:53:11.384 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:11.546 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (163 ms) 2023-10-18T03:53:11.547 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2023-10-18T03:53:11.547 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:11.718 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (172 ms) 2023-10-18T03:53:11.718 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2023-10-18T03:53:11.719 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:11.884 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (166 ms) 2023-10-18T03:53:11.884 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2023-10-18T03:53:11.885 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:12.044 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (160 ms) 2023-10-18T03:53:12.045 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2023-10-18T03:53:12.045 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:12.192 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:12.192 INFO:tasks.workunit.client.0.smithi003.stdout:aio discard: 0~1048576 = 0 2023-10-18T03:53:12.193 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:12.193 INFO:tasks.workunit.client.0.smithi003.stdout:aio discard: 0~4194304 = 0 2023-10-18T03:53:12.193 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:12.193 INFO:tasks.workunit.client.0.smithi003.stdout:aio discard: 3145728~1048576 = 0 2023-10-18T03:53:12.194 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (148 ms) 2023-10-18T03:53:12.194 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestFUA 2023-10-18T03:53:12.194 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:13.087 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.088 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.088 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.088 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.088 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.088 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.089 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.089 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.089 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.089 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.089 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.090 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.090 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.090 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.090 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.091 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.091 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.091 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.091 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.091 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.092 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.092 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.092 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.092 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.092 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.092 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.093 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.093 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.093 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.093 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.093 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.094 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.094 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.094 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.094 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.094 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.095 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.095 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.096 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.096 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.096 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.096 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.097 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.097 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.097 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.097 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.097 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.098 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.098 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.098 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.098 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.098 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.099 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.099 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.099 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.099 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestFUA (895 ms) 2023-10-18T03:53:13.099 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestIOPP 2023-10-18T03:53:13.100 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:13.199 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.199 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.199 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.200 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.200 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:13.200 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.200 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.200 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.201 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.201 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.201 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.201 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.201 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.201 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.202 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.202 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.202 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.202 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.202 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.203 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.203 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.203 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.203 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.203 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.204 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.204 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.204 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:13.204 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.289 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.289 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:13.290 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:53:13.290 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:53:13.290 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:53:13.290 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:53:13.290 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:53:13.291 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:53:13.291 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:53:13.291 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:53:13.291 INFO:tasks.workunit.client.0.smithi003.stdout:start compare and write 2023-10-18T03:53:13.291 INFO:tasks.workunit.client.0.smithi003.stdout:compare and wrote: 512 2023-10-18T03:53:13.292 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.292 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:53:13.292 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.296 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.296 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:53:13.296 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.297 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:53:13.297 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.303 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.303 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:53:13.303 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.304 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:53:13.304 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.310 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.311 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:53:13.311 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.311 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:53:13.311 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.317 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.318 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:53:13.318 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.318 INFO:tasks.workunit.client.0.smithi003.stdout:started aio compare and write 2023-10-18T03:53:13.318 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:13.330 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:13.330 INFO:tasks.workunit.client.0.smithi003.stdout:finished aio compare and write 2023-10-18T03:53:13.330 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.331 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.331 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.331 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.331 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:13.331 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.332 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.332 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.332 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.332 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:13.333 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.333 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.333 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.334 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.334 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:13.334 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.334 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.335 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.336 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.336 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:13.336 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.336 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.336 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.337 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.338 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:13.338 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:13.338 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:13.338 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:13.339 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:13.339 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:13.339 INFO:tasks.workunit.client.0.smithi003.stdout:discard: 512~512 2023-10-18T03:53:13.339 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:23.247 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:23.247 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:23.247 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:23.247 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:23.248 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:23.248 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:53:23.248 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.248 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.248 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:53:23.249 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.249 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.249 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:53:23.249 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.249 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.250 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:53:23.250 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.250 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.250 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.250 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.251 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:53:23.251 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.251 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.251 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:53:23.251 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.251 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.252 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:53:23.252 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.252 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.252 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:53:23.252 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.253 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.253 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.253 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.253 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:53:23.254 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.254 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.254 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:53:23.254 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.254 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.255 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:53:23.255 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.255 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.255 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:53:23.255 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.256 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.256 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.256 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.256 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:53:23.256 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.257 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.257 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:53:23.257 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.257 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.257 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:53:23.258 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.258 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.258 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:53:23.258 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.258 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.259 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.259 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.259 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:53:23.259 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.260 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.260 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:53:23.260 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.260 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.260 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:53:23.261 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.261 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.261 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:53:23.261 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.262 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.262 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.262 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.262 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.262 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.263 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:23.263 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:23.263 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:23.263 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.264 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.264 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.264 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.264 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:23.264 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:23.265 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:23.265 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.265 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.265 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.266 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.266 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:23.316 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:23.316 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:23.317 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.317 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.317 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.317 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.317 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:23.381 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:23.381 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:23.381 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.382 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.382 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.382 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.382 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.382 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.383 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.383 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:23.383 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.383 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.383 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:23.694 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:23.694 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:23.694 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.695 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.695 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.695 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.695 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:23.875 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:23.875 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:23.876 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:23.876 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:23.876 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:23.876 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:23.876 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:24.132 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:24.133 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:24.133 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:24.133 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:24.133 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:24.134 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:24.134 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:24.375 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:24.377 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:24.378 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:24.378 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:24.378 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:24.378 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:24.378 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:24.379 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:24.379 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:24.379 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:24.379 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:24.379 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:24.380 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:24.799 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:24.799 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:24.799 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:24.799 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:24.800 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:24.800 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:24.800 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:25.410 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:25.411 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:25.411 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:25.411 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:25.411 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:25.412 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:25.412 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:25.998 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:25.998 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:25.998 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:25.998 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:25.999 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:25.999 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:25.999 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:26.441 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:26.441 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:26.442 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:26.442 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:26.442 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:26.442 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:26.443 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:26.443 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:26.443 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:26.443 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:26.443 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:26.444 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:26.444 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:27.229 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:27.229 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:27.229 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:27.230 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:27.230 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:27.230 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:27.230 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:27.771 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:27.771 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:27.772 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:27.772 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:27.772 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:27.772 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:27.772 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:28.425 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:28.425 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:28.425 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:28.425 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:28.426 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:28.426 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:28.426 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:29.223 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:29.223 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:29.223 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:29.224 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:29.224 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:29.224 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:29.224 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:29.224 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:29.225 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:29.225 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:29.225 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:29.225 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:29.226 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:30.211 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:30.212 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:30.212 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:30.212 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:30.212 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:30.212 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:30.213 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:31.133 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:31.134 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:31.134 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:31.134 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:31.134 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:31.134 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:31.135 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:32.087 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:32.087 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:32.087 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:32.989 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:32.989 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:32.989 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:32.990 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:32.990 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:32.990 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:32.990 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:32.990 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:32.991 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:32.991 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:32.991 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestIOPP (19901 ms) 2023-10-18T03:53:32.991 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2023-10-18T03:53:32.991 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:33.167 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (179 ms) 2023-10-18T03:53:33.168 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2023-10-18T03:53:33.168 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:33.354 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (187 ms) 2023-10-18T03:53:33.355 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2023-10-18T03:53:33.355 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:33.505 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (151 ms) 2023-10-18T03:53:33.506 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2023-10-18T03:53:33.506 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:33.703 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (198 ms) 2023-10-18T03:53:33.704 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2023-10-18T03:53:33.704 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:33.966 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (263 ms) 2023-10-18T03:53:33.967 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2023-10-18T03:53:33.967 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:34.101 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:34.133 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (166 ms) 2023-10-18T03:53:34.133 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2023-10-18T03:53:34.133 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:34.310 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (178 ms) 2023-10-18T03:53:34.311 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2023-10-18T03:53:34.311 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:34.472 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:34.508 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (198 ms) 2023-10-18T03:53:34.509 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2023-10-18T03:53:34.509 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:34.810 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (301 ms) 2023-10-18T03:53:34.810 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2023-10-18T03:53:34.810 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:34.954 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:34.985 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (175 ms) 2023-10-18T03:53:34.985 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2023-10-18T03:53:34.985 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:35.157 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (172 ms) 2023-10-18T03:53:35.157 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2023-10-18T03:53:35.157 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:35.321 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:35.354 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (197 ms) 2023-10-18T03:53:35.354 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2023-10-18T03:53:35.354 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:35.609 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (256 ms) 2023-10-18T03:53:35.609 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2023-10-18T03:53:35.610 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:35.791 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:35.834 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (225 ms) 2023-10-18T03:53:35.834 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2023-10-18T03:53:35.834 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:36.066 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (232 ms) 2023-10-18T03:53:36.066 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2023-10-18T03:53:36.067 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:36.253 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:36.302 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (236 ms) 2023-10-18T03:53:36.303 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2023-10-18T03:53:36.303 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:37.569 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (1267 ms) 2023-10-18T03:53:37.570 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2023-10-18T03:53:37.570 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:38.912 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:38.973 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (1403 ms) 2023-10-18T03:53:38.973 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2023-10-18T03:53:38.973 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:53:39.142 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:39.143 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:39.143 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:39.143 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:39.143 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:53:39.144 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.144 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:39.144 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:39.148 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:39.149 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:39.149 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.149 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:39.149 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:39.155 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:39.155 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:39.156 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.156 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:39.156 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:39.162 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:39.162 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:39.163 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.163 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:39.163 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:39.171 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:39.172 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:39.172 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.172 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:53:39.172 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:39.176 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:39.176 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:53:39.176 INFO:tasks.workunit.client.0.smithi003.stdout:read: 512 2023-10-18T03:53:39.176 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.177 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:39.177 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:39.178 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:39.178 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:39.178 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.178 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:39.178 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:39.179 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:39.180 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:39.180 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.180 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:39.180 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:39.181 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:39.182 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:39.182 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.182 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:39.182 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:39.183 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:39.184 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:39.184 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:39.184 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:53:39.184 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:53:48.715 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:53:48.716 INFO:tasks.workunit.client.0.smithi003.stdout:finished read 2023-10-18T03:53:48.716 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:53:48.716 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.716 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.717 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 0 2023-10-18T03:53:48.717 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.717 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.717 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:53:48.717 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.718 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.718 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 16384 2023-10-18T03:53:48.718 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.718 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.718 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.719 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.719 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:53:48.719 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.719 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.719 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 49152 2023-10-18T03:53:48.720 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.720 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.720 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:53:48.720 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.720 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.721 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 65536 2023-10-18T03:53:48.721 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.721 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.721 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.721 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.722 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:53:48.722 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.722 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.722 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 98304 2023-10-18T03:53:48.722 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.723 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.723 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:53:48.723 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.723 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.724 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 114688 2023-10-18T03:53:48.724 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.724 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.724 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.724 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.724 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:53:48.725 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.725 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.725 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 147456 2023-10-18T03:53:48.725 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.725 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.726 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:53:48.726 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.726 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.726 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 163840 2023-10-18T03:53:48.726 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.727 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.727 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.727 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.727 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:53:48.727 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.728 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.728 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 196608 2023-10-18T03:53:48.728 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.728 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.728 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:53:48.729 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.729 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.729 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 212992 2023-10-18T03:53:48.729 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.729 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.729 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.730 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.730 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:48.730 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:48.730 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:48.730 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:48.731 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:48.731 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.731 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.731 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:48.731 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:48.732 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:48.732 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:48.732 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:48.732 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.733 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.733 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:48.733 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:48.733 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:48.827 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:48.827 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:48.827 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.828 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.828 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:48.829 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:48.829 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:48.967 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:48.967 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:48.968 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:48.968 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:48.968 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:48.968 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:48.968 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.969 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:48.969 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:48.969 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:48.969 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:48.969 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:48.970 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:49.196 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:49.196 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:49.197 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:49.197 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:49.197 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:49.198 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:49.198 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:49.421 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:49.421 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:49.421 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:49.421 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:49.422 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:49.422 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:49.422 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:49.716 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:49.717 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:49.717 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:49.717 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:49.717 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:49.717 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:49.718 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:50.058 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:50.058 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:50.059 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:50.059 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:50.059 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:50.059 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:50.060 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:50.060 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:50.060 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:50.060 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:50.060 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:50.061 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:50.061 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:50.467 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:50.467 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:50.468 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:50.468 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:50.468 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:50.468 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:50.469 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:51.020 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:51.020 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:51.020 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:51.020 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:51.021 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:51.021 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:51.021 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:51.499 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:51.500 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:51.500 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:51.500 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:51.500 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:51.501 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:51.501 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:51.924 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:51.925 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:51.925 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:51.925 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:51.925 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:51.926 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:51.926 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:51.926 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:51.926 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:51.926 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:51.927 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:51.927 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:51.927 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:52.530 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:52.530 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:52.530 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:52.530 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:52.531 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:52.531 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:52.531 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:53.238 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:53.238 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:53.238 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:53.239 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:53.239 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:53.239 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:53.239 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:54.068 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:54.068 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:54.068 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:54.068 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:54.069 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:54.069 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:54.069 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:54.904 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:54.905 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:54.905 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:54.905 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:54.905 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:54.905 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:54.906 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:54.906 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:54.906 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:54.906 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:54.906 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:54.907 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:54.907 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:55.778 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:55.778 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:55.778 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:55.778 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:55.779 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:55.779 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:55.779 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:56.488 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:56.488 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:56.488 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:56.489 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:56.489 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:56.489 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:56.489 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:57.499 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:57.499 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:57.500 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:57.500 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:57.500 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:57.500 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:57.500 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:53:58.449 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:53:58.450 INFO:tasks.workunit.client.0.smithi003.stdout:finished writesame 2023-10-18T03:53:58.450 INFO:tasks.workunit.client.0.smithi003.stdout:to verify the data 2023-10-18T03:53:58.450 INFO:tasks.workunit.client.0.smithi003.stdout:verified 2023-10-18T03:53:58.450 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:58.451 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:58.451 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:58.451 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:53:58.451 INFO:tasks.workunit.client.0.smithi003.stdout:started writesame 2023-10-18T03:53:58.451 INFO:tasks.workunit.client.0.smithi003.stdout:expected fail, finished writesame 2023-10-18T03:53:58.451 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (19477 ms) 2023-10-18T03:53:58.452 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2023-10-18T03:53:58.452 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:00.854 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 0 2023-10-18T03:54:00.855 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:54:00.855 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 1 2023-10-18T03:54:00.855 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:54:00.855 INFO:tasks.workunit.client.0.smithi003.stdout:snap: orig 2023-10-18T03:54:00.856 INFO:tasks.workunit.client.0.smithi003.stdout:found orig with size 2097152 2023-10-18T03:54:00.856 INFO:tasks.workunit.client.0.smithi003.stdout:read: 80 2023-10-18T03:54:00.856 INFO:tasks.workunit.client.0.smithi003.stdout:write test data! 2023-10-18T03:54:00.856 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 80 2023-10-18T03:54:00.856 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 2 2023-10-18T03:54:00.857 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:54:00.857 INFO:tasks.workunit.client.0.smithi003.stdout:snap: orig 2023-10-18T03:54:00.857 INFO:tasks.workunit.client.0.smithi003.stdout:snap: written 2023-10-18T03:54:00.857 INFO:tasks.workunit.client.0.smithi003.stdout:found orig with size 2097152 2023-10-18T03:54:00.857 INFO:tasks.workunit.client.0.smithi003.stdout:found written with size 2097152 2023-10-18T03:54:00.857 INFO:tasks.workunit.client.0.smithi003.stdout:read: 80 2023-10-18T03:54:00.858 INFO:tasks.workunit.client.0.smithi003.stdout:read: 80 2023-10-18T03:54:00.858 INFO:tasks.workunit.client.0.smithi003.stdout:read: 80 2023-10-18T03:54:00.858 INFO:tasks.workunit.client.0.smithi003.stdout:write to snapshot returned -30 2023-10-18T03:54:00.858 INFO:tasks.workunit.client.0.smithi003.stdout:Read-only file system 2023-10-18T03:54:01.029 INFO:tasks.workunit.client.0.smithi003.stdout:read: 80 2023-10-18T03:54:01.029 INFO:tasks.workunit.client.0.smithi003.stdout:read: 80 2023-10-18T03:54:01.030 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 80 2023-10-18T03:54:01.030 INFO:tasks.workunit.client.0.smithi003.stdout:opening testimg@orig 2023-10-18T03:54:01.030 INFO:tasks.workunit.client.0.smithi003.stdout:read: 80 2023-10-18T03:54:01.030 INFO:tasks.workunit.client.0.smithi003.stdout:write to snapshot returned -30 2023-10-18T03:54:01.030 INFO:tasks.workunit.client.0.smithi003.stdout:Read-only file system 2023-10-18T03:54:02.671 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 2 2023-10-18T03:54:02.671 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:54:02.671 INFO:tasks.workunit.client.0.smithi003.stdout:snap: orig 2023-10-18T03:54:02.671 INFO:tasks.workunit.client.0.smithi003.stdout:snap: written 2023-10-18T03:54:02.672 INFO:tasks.workunit.client.0.smithi003.stdout:found orig with size 2097152 2023-10-18T03:54:02.672 INFO:tasks.workunit.client.0.smithi003.stdout:found written with size 2097152 2023-10-18T03:54:02.672 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 1 2023-10-18T03:54:02.672 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:54:02.672 INFO:tasks.workunit.client.0.smithi003.stdout:snap: orig 2023-10-18T03:54:02.672 INFO:tasks.workunit.client.0.smithi003.stdout:found orig with size 2097152 2023-10-18T03:54:02.673 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 0 2023-10-18T03:54:02.673 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:54:02.673 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (4221 ms) 2023-10-18T03:54:02.673 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2023-10-18T03:54:02.673 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:04.533 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (1863 ms) 2023-10-18T03:54:04.533 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestClone 2023-10-18T03:54:04.534 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:11.526 INFO:tasks.workunit.client.0.smithi003.stdout:made parent image "parent" 2023-10-18T03:54:11.527 INFO:tasks.workunit.client.0.smithi003.stdout:parent has no parent info 2023-10-18T03:54:11.527 INFO:tasks.workunit.client.0.smithi003.stdout:made snapshot "parent@parent_snap" 2023-10-18T03:54:11.527 INFO:tasks.workunit.client.0.smithi003.stdout:can't unprotect an unprotected snap 2023-10-18T03:54:11.527 INFO:tasks.workunit.client.0.smithi003.stdout:can't protect a protected snap 2023-10-18T03:54:11.528 INFO:tasks.workunit.client.0.smithi003.stdout:made and opened clone "child" 2023-10-18T03:54:11.528 INFO:tasks.workunit.client.0.smithi003.stdout:read: 8 2023-10-18T03:54:11.528 INFO:tasks.workunit.client.0.smithi003.stdout:read: 8 2023-10-18T03:54:11.528 INFO:tasks.workunit.client.0.smithi003.stdout:read: 8 2023-10-18T03:54:11.528 INFO:tasks.workunit.client.0.smithi003.stdout:sizes and overlaps are good between parent and child 2023-10-18T03:54:11.529 INFO:tasks.workunit.client.0.smithi003.stdout:child image successfully cloned all image-meta pairs 2023-10-18T03:54:11.529 INFO:tasks.workunit.client.0.smithi003.stdout:sized down clone, changed overlap 2023-10-18T03:54:11.529 INFO:tasks.workunit.client.0.smithi003.stdout:parent info: size 4194304 obj_size 4194304 parent_pool 18446744073709551615 2023-10-18T03:54:11.529 INFO:tasks.workunit.client.0.smithi003.stdout:sized up clone, changed size but not overlap or parent's size 2023-10-18T03:54:11.529 INFO:tasks.workunit.client.0.smithi003.stdout:can't remove parent while child still exists 2023-10-18T03:54:11.530 INFO:tasks.workunit.client.0.smithi003.stdout:can't remove parent while still protected 2023-10-18T03:54:11.530 INFO:tasks.workunit.client.0.smithi003.stdout:removed parent snap after unprotecting 2023-10-18T03:54:11.530 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestClone (6993 ms) 2023-10-18T03:54:11.530 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestClone2 2023-10-18T03:54:11.530 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:16.406 INFO:tasks.workunit.client.0.smithi003.stdout:made parent image "parent" 2023-10-18T03:54:16.406 INFO:tasks.workunit.client.0.smithi003.stdout:parent has no parent info 2023-10-18T03:54:16.407 INFO:tasks.workunit.client.0.smithi003.stdout:made snapshot "parent@parent_snap" 2023-10-18T03:54:16.407 INFO:tasks.workunit.client.0.smithi003.stdout:made and opened clone "child" 2023-10-18T03:54:16.407 INFO:tasks.workunit.client.0.smithi003.stdout:child image successfully cloned all image-meta pairs 2023-10-18T03:54:16.407 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestClone2 (4880 ms) 2023-10-18T03:54:16.407 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ListChildren 2023-10-18T03:54:19.994 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:29.471 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.471 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.471 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.472 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.472 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.472 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 1 2023-10-18T03:54:29.472 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:54:29.472 INFO:tasks.workunit.client.0.smithi003.stdout:child: image113 2023-10-18T03:54:29.473 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113/105c5ede0dcc 2023-10-18T03:54:29.473 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.473 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.473 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.473 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.474 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.474 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.474 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.474 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.474 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.475 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.475 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.475 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.475 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 2 2023-10-18T03:54:29.475 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:54:29.476 INFO:tasks.workunit.client.0.smithi003.stdout:child: image114 2023-10-18T03:54:29.476 INFO:tasks.workunit.client.0.smithi003.stdout:child: image113 2023-10-18T03:54:29.476 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113/105c5ede0dcc 2023-10-18T03:54:29.476 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.476 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114/105cb80cedf8 2023-10-18T03:54:29.477 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.477 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.477 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.477 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.477 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.478 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.478 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.478 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.478 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.478 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.479 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.479 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.479 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.479 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:29.479 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.480 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.480 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:29.480 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:29.480 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.480 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 3 2023-10-18T03:54:29.481 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 3 2023-10-18T03:54:29.481 INFO:tasks.workunit.client.0.smithi003.stdout:child: image114 2023-10-18T03:54:29.481 INFO:tasks.workunit.client.0.smithi003.stdout:child: image113 2023-10-18T03:54:29.481 INFO:tasks.workunit.client.0.smithi003.stdout:child: image115 2023-10-18T03:54:29.481 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113/105c5ede0dcc 2023-10-18T03:54:29.481 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114/105cb80cedf8 2023-10-18T03:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image115/105cd2c4b5 2023-10-18T03:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 3 2023-10-18T03:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 3 2023-10-18T03:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:child: image114 2023-10-18T03:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:child: image113 2023-10-18T03:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:child: image115 2023-10-18T03:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113/105c5ede0dcc 2023-10-18T03:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114/105cb80cedf8 2023-10-18T03:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image115/105cd2c4b5 2023-10-18T03:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.490 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.490 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:29.490 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:29.490 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.490 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 4 2023-10-18T03:54:29.491 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 4 2023-10-18T03:54:29.491 INFO:tasks.workunit.client.0.smithi003.stdout:child: image114 2023-10-18T03:54:29.491 INFO:tasks.workunit.client.0.smithi003.stdout:child: image113 2023-10-18T03:54:29.491 INFO:tasks.workunit.client.0.smithi003.stdout:child: image115 2023-10-18T03:54:29.491 INFO:tasks.workunit.client.0.smithi003.stdout:child: image116 2023-10-18T03:54:29.492 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113/105c5ede0dcc 2023-10-18T03:54:29.492 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.492 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114/105cb80cedf8 2023-10-18T03:54:29.492 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.492 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image115/105cd2c4b5 2023-10-18T03:54:29.492 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.493 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image116/105ca063c81e 2023-10-18T03:54:29.493 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.493 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.493 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.493 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.494 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.494 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.494 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.494 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.494 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.495 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.495 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.495 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.495 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.495 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:29.496 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:29.496 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:29.496 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:29.496 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:29.496 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.496 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:29.497 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:29.497 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.168 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image113 2023-10-18T03:54:34.168 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:34.169 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:34.169 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:34.169 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.169 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 4 2023-10-18T03:54:34.170 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 4 2023-10-18T03:54:34.170 INFO:tasks.workunit.client.0.smithi003.stdout:child: image114 2023-10-18T03:54:34.170 INFO:tasks.workunit.client.0.smithi003.stdout:child: image113 2023-10-18T03:54:34.170 INFO:tasks.workunit.client.0.smithi003.stdout:child: image115 2023-10-18T03:54:34.170 INFO:tasks.workunit.client.0.smithi003.stdout:child: image116 2023-10-18T03:54:34.171 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image113/105c5ede0dcc 2023-10-18T03:54:34.171 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.171 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114/105cb80cedf8 2023-10-18T03:54:34.171 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.171 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image115/105cd2c4b5 2023-10-18T03:54:34.172 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.172 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image116/105ca063c81e 2023-10-18T03:54:34.172 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.172 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.172 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.173 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.173 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.173 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.173 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.173 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:34.174 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.174 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:34.174 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:34.174 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.174 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.175 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:34.175 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.175 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image115 2023-10-18T03:54:34.175 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:34.175 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:34.176 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.176 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 3 2023-10-18T03:54:34.176 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 3 2023-10-18T03:54:34.176 INFO:tasks.workunit.client.0.smithi003.stdout:child: image114 2023-10-18T03:54:34.176 INFO:tasks.workunit.client.0.smithi003.stdout:child: image115 2023-10-18T03:54:34.177 INFO:tasks.workunit.client.0.smithi003.stdout:child: image116 2023-10-18T03:54:34.177 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114/105cb80cedf8 2023-10-18T03:54:34.177 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.177 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image115/105cd2c4b5 2023-10-18T03:54:34.178 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.178 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image116/105ca063c81e 2023-10-18T03:54:34.178 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.178 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.179 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.179 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.179 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.180 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.180 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.180 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:34.180 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.180 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:34.181 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image116 2023-10-18T03:54:34.181 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.181 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 2 2023-10-18T03:54:34.181 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 2 2023-10-18T03:54:34.181 INFO:tasks.workunit.client.0.smithi003.stdout:child: image114 2023-10-18T03:54:34.181 INFO:tasks.workunit.client.0.smithi003.stdout:child: image116 2023-10-18T03:54:34.182 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114/105cb80cedf8 2023-10-18T03:54:34.182 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.182 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-14/image116/105ca063c81e 2023-10-18T03:54:34.182 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.182 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.183 INFO:tasks.workunit.client.0.smithi003.stdout:trying to find test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.183 INFO:tasks.workunit.client.0.smithi003.stdout:checking test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.183 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114 2023-10-18T03:54:34.183 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.183 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 1 2023-10-18T03:54:34.184 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 1 2023-10-18T03:54:34.184 INFO:tasks.workunit.client.0.smithi003.stdout:child: image114 2023-10-18T03:54:34.184 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-13/image114/105cb80cedf8 2023-10-18T03:54:34.184 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:54:34.184 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 0 2023-10-18T03:54:34.185 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:54:34.185 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ListChildren (17761 ms) 2023-10-18T03:54:34.185 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2023-10-18T03:54:34.185 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_librbd.cc:6928: Skipped 2023-10-18T03:54:34.186 INFO:tasks.workunit.client.0.smithi003.stdout:Not supported by crimson yet. Skipped 2023-10-18T03:54:34.187 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered (0 ms) 2023-10-18T03:54:34.187 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.LockingPP 2023-10-18T03:54:34.188 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:34.519 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.LockingPP (352 ms) 2023-10-18T03:54:34.520 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.FlushAio 2023-10-18T03:54:34.520 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:37.918 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.FlushAio (3399 ms) 2023-10-18T03:54:37.919 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.FlushAioPP 2023-10-18T03:54:37.919 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:39.373 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.FlushAioPP (1455 ms) 2023-10-18T03:54:39.373 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapDiff 2023-10-18T03:54:39.374 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:45.313 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:54:45.314 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapDiff (5940 ms) 2023-10-18T03:54:45.314 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2023-10-18T03:54:45.314 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:45.477 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (163 ms) 2023-10-18T03:54:45.477 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2023-10-18T03:54:45.477 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:45.713 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (237 ms) 2023-10-18T03:54:45.713 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2023-10-18T03:54:45.713 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:45.862 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ZeroLengthRead (148 ms) 2023-10-18T03:54:45.862 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2023-10-18T03:54:45.863 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:46.065 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.LargeCacheRead (204 ms) 2023-10-18T03:54:46.066 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestPendingAio 2023-10-18T03:54:46.066 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:48.279 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestPendingAio (2214 ms) 2023-10-18T03:54:48.279 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2023-10-18T03:54:48.280 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:50.286 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:53.943 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyup (5664 ms) 2023-10-18T03:54:53.944 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2023-10-18T03:54:53.944 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:55.359 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:54:59.126 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyupMismatch (5182 ms) 2023-10-18T03:54:59.126 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.Flatten 2023-10-18T03:54:59.126 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:01.365 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.Flatten (2240 ms) 2023-10-18T03:55:01.366 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.Sparsify 2023-10-18T03:55:01.366 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:01.595 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 32768 2023-10-18T03:55:01.596 INFO:tasks.workunit.client.0.smithi003.stdout:read: 32768 2023-10-18T03:55:01.596 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.Sparsify (230 ms) 2023-10-18T03:55:01.596 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SparsifyPP 2023-10-18T03:55:01.596 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:01.982 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SparsifyPP (387 ms) 2023-10-18T03:55:01.983 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2023-10-18T03:55:01.983 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:07.829 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapshotLimit (5847 ms) 2023-10-18T03:55:07.829 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2023-10-18T03:55:07.830 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:12.397 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (4567 ms) 2023-10-18T03:55:12.397 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2023-10-18T03:55:12.397 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:13.130 INFO:tasks.workunit.client.0.smithi003.stdout:0.00% done 2023-10-18T03:55:13.130 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.RebuildObjectMapViaLockOwner (733 ms) 2023-10-18T03:55:13.130 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2023-10-18T03:55:13.131 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:13.604 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.RenameViaLockOwner (474 ms) 2023-10-18T03:55:13.604 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2023-10-18T03:55:13.604 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:15.669 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapCreateViaLockOwner (2066 ms) 2023-10-18T03:55:15.670 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2023-10-18T03:55:15.670 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:18.797 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapRemoveViaLockOwner (3127 ms) 2023-10-18T03:55:18.797 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2023-10-18T03:55:18.798 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:19.199 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.UpdateFeaturesViaLockOwner (402 ms) 2023-10-18T03:55:19.199 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2023-10-18T03:55:19.200 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_librbd.cc:8745: Skipped 2023-10-18T03:55:19.200 INFO:tasks.workunit.client.0.smithi003.stdout:Skipping due to unmet REQUIRE 2023-10-18T03:55:19.200 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2023-10-18T03:55:19.200 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapRemove2 2023-10-18T03:55:19.200 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:22.733 INFO:tasks.workunit.client.0.smithi003.stdout:100.00% done 2023-10-18T03:55:22.734 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapRemove2 (3535 ms) 2023-10-18T03:55:22.734 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2023-10-18T03:55:22.734 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:24.822 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapRenameViaLockOwner (2088 ms) 2023-10-18T03:55:24.822 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2023-10-18T03:55:24.823 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:26.045 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapProtectViaLockOwner (1223 ms) 2023-10-18T03:55:26.045 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2023-10-18T03:55:26.046 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:27.857 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapUnprotectViaLockOwner (1811 ms) 2023-10-18T03:55:27.857 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2023-10-18T03:55:27.857 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:29.363 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.FlattenViaLockOwner (1506 ms) 2023-10-18T03:55:29.363 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2023-10-18T03:55:29.364 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:29.658 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ResizeViaLockOwner (295 ms) 2023-10-18T03:55:29.659 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2023-10-18T03:55:29.659 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:30.044 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SparsifyViaLockOwner (386 ms) 2023-10-18T03:55:30.044 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2023-10-18T03:55:30.045 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:47.213 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ObjectMapConsistentSnap (17169 ms) 2023-10-18T03:55:47.214 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.Metadata 2023-10-18T03:55:47.214 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:49.318 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.Metadata (2104 ms) 2023-10-18T03:55:49.318 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.MetadataPP 2023-10-18T03:55:49.318 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:51.359 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.MetadataPP (2042 ms) 2023-10-18T03:55:51.360 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2023-10-18T03:55:51.360 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:52.250 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.UpdateFeatures (890 ms) 2023-10-18T03:55:52.250 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2023-10-18T03:55:52.250 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2023-10-18T03:55:52.250 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2023-10-18T03:55:52.251 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:54.083 INFO:tasks.workunit.client.0.smithi003.stdout:0.00% done 2023-10-18T03:55:54.083 INFO:tasks.workunit.client.0.smithi003.stdout:25.00% done 2023-10-18T03:55:54.083 INFO:tasks.workunit.client.0.smithi003.stdout:50.00% done 2023-10-18T03:55:54.083 INFO:tasks.workunit.client.0.smithi003.stdout:75.00% done 2023-10-18T03:55:54.084 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.RebuildObjectMap (1833 ms) 2023-10-18T03:55:54.084 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2023-10-18T03:55:54.315 INFO:tasks.workunit.client.0.smithi003.stdout:0.00% done 2023-10-18T03:55:54.315 INFO:tasks.workunit.client.0.smithi003.stdout:25.00% done 2023-10-18T03:55:54.316 INFO:tasks.workunit.client.0.smithi003.stdout:50.00% done 2023-10-18T03:55:54.316 INFO:tasks.workunit.client.0.smithi003.stdout:75.00% done 2023-10-18T03:55:54.316 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.RebuildNewObjectMap (233 ms) 2023-10-18T03:55:54.316 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2023-10-18T03:55:54.316 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:56.028 INFO:tasks.workunit.client.0.smithi003.stdout:0.00% done 2023-10-18T03:55:56.028 INFO:tasks.workunit.client.0.smithi003.stdout:25.00% done 2023-10-18T03:55:56.029 INFO:tasks.workunit.client.0.smithi003.stdout:50.00% done 2023-10-18T03:55:56.029 INFO:tasks.workunit.client.0.smithi003.stdout:75.00% done 2023-10-18T03:55:56.029 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.CheckObjectMap (1713 ms) 2023-10-18T03:55:56.029 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.BlockingAIO 2023-10-18T03:55:56.029 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:56.200 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.BlockingAIO (171 ms) 2023-10-18T03:55:56.200 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2023-10-18T03:55:56.200 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:57.615 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ExclusiveLockTransition (1416 ms) 2023-10-18T03:55:57.616 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2023-10-18T03:55:57.616 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_librbd.cc:9802: Skipped 2023-10-18T03:55:57.616 INFO:tasks.workunit.client.0.smithi003.stdout:Skipping due to unmet REQUIRE 2023-10-18T03:55:57.616 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2023-10-18T03:55:57.616 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2023-10-18T03:55:57.617 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:55:59.140 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.CacheMayCopyOnWrite (1525 ms) 2023-10-18T03:55:59.141 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2023-10-18T03:55:59.141 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:00.870 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (1730 ms) 2023-10-18T03:56:00.871 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestImageOptions 2023-10-18T03:56:03.312 INFO:tasks.workunit.client.0.smithi003.stdout:0.00% done 2023-10-18T03:56:03.312 INFO:tasks.workunit.client.0.smithi003.stdout:100.00% done 2023-10-18T03:56:03.313 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestImageOptions (2441 ms) 2023-10-18T03:56:03.313 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2023-10-18T03:56:05.298 INFO:tasks.workunit.client.0.smithi003.stdout:0.00% done 2023-10-18T03:56:05.298 INFO:tasks.workunit.client.0.smithi003.stdout:100.00% done 2023-10-18T03:56:05.298 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (1986 ms) 2023-10-18T03:56:05.299 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2023-10-18T03:56:05.299 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2023-10-18T03:56:05.299 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2023-10-18T03:56:05.299 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2023-10-18T03:56:05.299 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ImagePollIO 2023-10-18T03:56:05.300 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:05.612 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:56:05.613 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:56:05.613 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:56:05.613 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:56:05.613 INFO:tasks.workunit.client.0.smithi003.stdout:wrote: 512 2023-10-18T03:56:05.613 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.614 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:56:05.614 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:56:05.614 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:56:05.614 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:56:05.614 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.615 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:56:05.615 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:56:05.615 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:56:05.615 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:56:05.615 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.616 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:56:05.616 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:56:05.616 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:56:05.617 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:56:05.617 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.617 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:56:05.617 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:56:05.618 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:56:05.618 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:56:05.618 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.618 INFO:tasks.workunit.client.0.smithi003.stdout:started write 2023-10-18T03:56:05.618 INFO:tasks.workunit.client.0.smithi003.stdout:write completion cb called! 2023-10-18T03:56:05.619 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 0 2023-10-18T03:56:05.619 INFO:tasks.workunit.client.0.smithi003.stdout:finished write 2023-10-18T03:56:05.619 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.619 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:56:05.619 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:56:05.620 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:56:05.620 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.620 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:56:05.620 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:56:05.621 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:56:05.621 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.621 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:56:05.621 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:56:05.621 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:56:05.622 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.622 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:56:05.622 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:56:05.622 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:56:05.622 INFO:tasks.workunit.client.0.smithi003.stdout:created completion 2023-10-18T03:56:05.623 INFO:tasks.workunit.client.0.smithi003.stdout:started read 2023-10-18T03:56:05.623 INFO:tasks.workunit.client.0.smithi003.stdout:read completion cb called! 2023-10-18T03:56:05.623 INFO:tasks.workunit.client.0.smithi003.stdout:return value is: 512 2023-10-18T03:56:05.623 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ImagePollIO (315 ms) 2023-10-18T03:56:05.623 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.Mirror 2023-10-18T03:56:05.712 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:05.799 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:07.717 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.Mirror (2105 ms) 2023-10-18T03:56:07.717 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2023-10-18T03:56:07.788 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (71 ms) 2023-10-18T03:56:07.789 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2023-10-18T03:56:15.814 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.CreateWithMirrorEnabled (8025 ms) 2023-10-18T03:56:15.817 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2023-10-18T03:56:15.818 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:18.603 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (2790 ms) 2023-10-18T03:56:18.604 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2023-10-18T03:56:18.604 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:19.354 INFO:tasks.workunit.client.0.smithi003.stdout:1: acquiring exclusive lock 2023-10-18T03:56:19.355 INFO:tasks.workunit.client.0.smithi003.stdout:2: acquiring exclusive lock 2023-10-18T03:56:19.386 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock acquired 2023-10-18T03:56:19.387 INFO:tasks.workunit.client.0.smithi003.stdout:2: releasing exclusive lock 2023-10-18T03:56:19.414 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock released 2023-10-18T03:56:19.414 INFO:tasks.workunit.client.0.smithi003.stdout:2: acquiring exclusive lock 2023-10-18T03:56:19.448 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock acquired 2023-10-18T03:56:19.456 INFO:tasks.workunit.client.0.smithi003.stdout:2: releasing exclusive lock 2023-10-18T03:56:19.483 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock released 2023-10-18T03:56:19.483 INFO:tasks.workunit.client.0.smithi003.stdout:2: acquiring exclusive lock 2023-10-18T03:56:19.514 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock acquired 2023-10-18T03:56:19.555 INFO:tasks.workunit.client.0.smithi003.stdout:1: releasing exclusive lock 2023-10-18T03:56:19.580 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock released 2023-10-18T03:56:19.580 INFO:tasks.workunit.client.0.smithi003.stdout:1: acquiring exclusive lock 2023-10-18T03:56:19.612 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock acquired 2023-10-18T03:56:19.637 INFO:tasks.workunit.client.0.smithi003.stdout:2: releasing exclusive lock 2023-10-18T03:56:19.712 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock released 2023-10-18T03:56:19.718 INFO:tasks.workunit.client.0.smithi003.stdout:2: acquiring exclusive lock 2023-10-18T03:56:19.776 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock acquired 2023-10-18T03:56:19.790 INFO:tasks.workunit.client.0.smithi003.stdout:2: releasing exclusive lock 2023-10-18T03:56:19.817 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock released 2023-10-18T03:56:19.817 INFO:tasks.workunit.client.0.smithi003.stdout:2: acquiring exclusive lock 2023-10-18T03:56:19.848 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock acquired 2023-10-18T03:56:19.889 INFO:tasks.workunit.client.0.smithi003.stdout:1: releasing exclusive lock 2023-10-18T03:56:19.912 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock released 2023-10-18T03:56:19.913 INFO:tasks.workunit.client.0.smithi003.stdout:1: acquiring exclusive lock 2023-10-18T03:56:19.943 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock acquired 2023-10-18T03:56:19.968 INFO:tasks.workunit.client.0.smithi003.stdout:2: releasing exclusive lock 2023-10-18T03:56:19.994 INFO:tasks.workunit.client.0.smithi003.stdout:2: exclusive lock released 2023-10-18T03:56:20.027 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock acquired 2023-10-18T03:56:20.055 INFO:tasks.workunit.client.0.smithi003.stdout:1: releasing exclusive lock 2023-10-18T03:56:20.077 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock released 2023-10-18T03:56:20.077 INFO:tasks.workunit.client.0.smithi003.stdout:1: acquiring exclusive lock 2023-10-18T03:56:20.105 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock acquired 2023-10-18T03:56:20.120 INFO:tasks.workunit.client.0.smithi003.stdout:1: releasing exclusive lock 2023-10-18T03:56:20.142 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock released 2023-10-18T03:56:20.143 INFO:tasks.workunit.client.0.smithi003.stdout:1: acquiring exclusive lock 2023-10-18T03:56:20.170 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock acquired 2023-10-18T03:56:20.205 INFO:tasks.workunit.client.0.smithi003.stdout:1: releasing exclusive lock 2023-10-18T03:56:20.227 INFO:tasks.workunit.client.0.smithi003.stdout:1: exclusive lock released 2023-10-18T03:56:20.345 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ExclusiveLock (1741 ms) 2023-10-18T03:56:20.345 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.BreakLock 2023-10-18T03:56:20.345 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_librbd.cc:10562: Skipped 2023-10-18T03:56:20.346 INFO:tasks.workunit.client.0.smithi003.stdout:Not supported by crimson yet. Skipped 2023-10-18T03:56:20.346 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.BreakLock (0 ms) 2023-10-18T03:56:20.346 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2023-10-18T03:56:20.346 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:20.488 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (144 ms) 2023-10-18T03:56:20.489 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2023-10-18T03:56:20.489 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2023-10-18T03:56:20.489 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2023-10-18T03:56:20.490 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:21.022 INFO:tasks.workunit.client.0.smithi003.stdout:0.00% done 2023-10-18T03:56:21.022 INFO:tasks.workunit.client.0.smithi003.stdout:1.56% done 2023-10-18T03:56:21.022 INFO:tasks.workunit.client.0.smithi003.stdout:3.12% done 2023-10-18T03:56:21.022 INFO:tasks.workunit.client.0.smithi003.stdout:4.69% done 2023-10-18T03:56:21.023 INFO:tasks.workunit.client.0.smithi003.stdout:6.25% done 2023-10-18T03:56:21.023 INFO:tasks.workunit.client.0.smithi003.stdout:7.81% done 2023-10-18T03:56:21.023 INFO:tasks.workunit.client.0.smithi003.stdout:9.38% done 2023-10-18T03:56:21.023 INFO:tasks.workunit.client.0.smithi003.stdout:10.94% done 2023-10-18T03:56:21.023 INFO:tasks.workunit.client.0.smithi003.stdout:12.50% done 2023-10-18T03:56:21.023 INFO:tasks.workunit.client.0.smithi003.stdout:14.06% done 2023-10-18T03:56:21.024 INFO:tasks.workunit.client.0.smithi003.stdout:15.62% done 2023-10-18T03:56:21.024 INFO:tasks.workunit.client.0.smithi003.stdout:17.19% done 2023-10-18T03:56:21.024 INFO:tasks.workunit.client.0.smithi003.stdout:18.75% done 2023-10-18T03:56:21.024 INFO:tasks.workunit.client.0.smithi003.stdout:20.31% done 2023-10-18T03:56:21.024 INFO:tasks.workunit.client.0.smithi003.stdout:21.88% done 2023-10-18T03:56:21.025 INFO:tasks.workunit.client.0.smithi003.stdout:23.44% done 2023-10-18T03:56:21.025 INFO:tasks.workunit.client.0.smithi003.stdout:25.00% done 2023-10-18T03:56:21.025 INFO:tasks.workunit.client.0.smithi003.stdout:26.56% done 2023-10-18T03:56:21.025 INFO:tasks.workunit.client.0.smithi003.stdout:28.12% done 2023-10-18T03:56:21.025 INFO:tasks.workunit.client.0.smithi003.stdout:29.69% done 2023-10-18T03:56:21.026 INFO:tasks.workunit.client.0.smithi003.stdout:31.25% done 2023-10-18T03:56:21.026 INFO:tasks.workunit.client.0.smithi003.stdout:32.81% done 2023-10-18T03:56:21.026 INFO:tasks.workunit.client.0.smithi003.stdout:34.38% done 2023-10-18T03:56:21.026 INFO:tasks.workunit.client.0.smithi003.stdout:35.94% done 2023-10-18T03:56:21.026 INFO:tasks.workunit.client.0.smithi003.stdout:37.50% done 2023-10-18T03:56:21.027 INFO:tasks.workunit.client.0.smithi003.stdout:39.06% done 2023-10-18T03:56:21.027 INFO:tasks.workunit.client.0.smithi003.stdout:40.62% done 2023-10-18T03:56:21.027 INFO:tasks.workunit.client.0.smithi003.stdout:42.19% done 2023-10-18T03:56:21.027 INFO:tasks.workunit.client.0.smithi003.stdout:43.75% done 2023-10-18T03:56:21.027 INFO:tasks.workunit.client.0.smithi003.stdout:45.31% done 2023-10-18T03:56:21.028 INFO:tasks.workunit.client.0.smithi003.stdout:46.88% done 2023-10-18T03:56:21.028 INFO:tasks.workunit.client.0.smithi003.stdout:48.44% done 2023-10-18T03:56:21.028 INFO:tasks.workunit.client.0.smithi003.stdout:50.00% done 2023-10-18T03:56:21.028 INFO:tasks.workunit.client.0.smithi003.stdout:51.56% done 2023-10-18T03:56:21.028 INFO:tasks.workunit.client.0.smithi003.stdout:53.12% done 2023-10-18T03:56:21.028 INFO:tasks.workunit.client.0.smithi003.stdout:54.69% done 2023-10-18T03:56:21.029 INFO:tasks.workunit.client.0.smithi003.stdout:56.25% done 2023-10-18T03:56:21.029 INFO:tasks.workunit.client.0.smithi003.stdout:57.81% done 2023-10-18T03:56:21.029 INFO:tasks.workunit.client.0.smithi003.stdout:59.38% done 2023-10-18T03:56:21.029 INFO:tasks.workunit.client.0.smithi003.stdout:60.94% done 2023-10-18T03:56:21.029 INFO:tasks.workunit.client.0.smithi003.stdout:62.50% done 2023-10-18T03:56:21.030 INFO:tasks.workunit.client.0.smithi003.stdout:64.06% done 2023-10-18T03:56:21.030 INFO:tasks.workunit.client.0.smithi003.stdout:65.62% done 2023-10-18T03:56:21.030 INFO:tasks.workunit.client.0.smithi003.stdout:67.19% done 2023-10-18T03:56:21.030 INFO:tasks.workunit.client.0.smithi003.stdout:68.75% done 2023-10-18T03:56:21.030 INFO:tasks.workunit.client.0.smithi003.stdout:70.31% done 2023-10-18T03:56:21.031 INFO:tasks.workunit.client.0.smithi003.stdout:71.88% done 2023-10-18T03:56:21.031 INFO:tasks.workunit.client.0.smithi003.stdout:73.44% done 2023-10-18T03:56:21.031 INFO:tasks.workunit.client.0.smithi003.stdout:75.00% done 2023-10-18T03:56:21.031 INFO:tasks.workunit.client.0.smithi003.stdout:76.56% done 2023-10-18T03:56:21.031 INFO:tasks.workunit.client.0.smithi003.stdout:78.12% done 2023-10-18T03:56:21.032 INFO:tasks.workunit.client.0.smithi003.stdout:79.69% done 2023-10-18T03:56:21.032 INFO:tasks.workunit.client.0.smithi003.stdout:81.25% done 2023-10-18T03:56:21.032 INFO:tasks.workunit.client.0.smithi003.stdout:82.81% done 2023-10-18T03:56:21.032 INFO:tasks.workunit.client.0.smithi003.stdout:84.38% done 2023-10-18T03:56:21.032 INFO:tasks.workunit.client.0.smithi003.stdout:85.94% done 2023-10-18T03:56:21.032 INFO:tasks.workunit.client.0.smithi003.stdout:87.50% done 2023-10-18T03:56:21.033 INFO:tasks.workunit.client.0.smithi003.stdout:89.06% done 2023-10-18T03:56:21.033 INFO:tasks.workunit.client.0.smithi003.stdout:90.62% done 2023-10-18T03:56:21.033 INFO:tasks.workunit.client.0.smithi003.stdout:92.19% done 2023-10-18T03:56:21.033 INFO:tasks.workunit.client.0.smithi003.stdout:93.75% done 2023-10-18T03:56:21.034 INFO:tasks.workunit.client.0.smithi003.stdout:95.31% done 2023-10-18T03:56:21.034 INFO:tasks.workunit.client.0.smithi003.stdout:96.88% done 2023-10-18T03:56:21.034 INFO:tasks.workunit.client.0.smithi003.stdout:98.44% done 2023-10-18T03:56:21.034 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurge (533 ms) 2023-10-18T03:56:21.034 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2023-10-18T03:56:21.035 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:21.507 INFO:tasks.workunit.client.0.smithi003.stdout:0.00% done 2023-10-18T03:56:21.507 INFO:tasks.workunit.client.0.smithi003.stdout:1.56% done 2023-10-18T03:56:21.508 INFO:tasks.workunit.client.0.smithi003.stdout:3.12% done 2023-10-18T03:56:21.508 INFO:tasks.workunit.client.0.smithi003.stdout:4.69% done 2023-10-18T03:56:21.508 INFO:tasks.workunit.client.0.smithi003.stdout:6.25% done 2023-10-18T03:56:21.508 INFO:tasks.workunit.client.0.smithi003.stdout:7.81% done 2023-10-18T03:56:21.508 INFO:tasks.workunit.client.0.smithi003.stdout:9.38% done 2023-10-18T03:56:21.509 INFO:tasks.workunit.client.0.smithi003.stdout:10.94% done 2023-10-18T03:56:21.509 INFO:tasks.workunit.client.0.smithi003.stdout:12.50% done 2023-10-18T03:56:21.509 INFO:tasks.workunit.client.0.smithi003.stdout:14.06% done 2023-10-18T03:56:21.509 INFO:tasks.workunit.client.0.smithi003.stdout:15.62% done 2023-10-18T03:56:21.509 INFO:tasks.workunit.client.0.smithi003.stdout:17.19% done 2023-10-18T03:56:21.510 INFO:tasks.workunit.client.0.smithi003.stdout:18.75% done 2023-10-18T03:56:21.510 INFO:tasks.workunit.client.0.smithi003.stdout:20.31% done 2023-10-18T03:56:21.510 INFO:tasks.workunit.client.0.smithi003.stdout:21.88% done 2023-10-18T03:56:21.510 INFO:tasks.workunit.client.0.smithi003.stdout:23.44% done 2023-10-18T03:56:21.510 INFO:tasks.workunit.client.0.smithi003.stdout:25.00% done 2023-10-18T03:56:21.511 INFO:tasks.workunit.client.0.smithi003.stdout:26.56% done 2023-10-18T03:56:21.511 INFO:tasks.workunit.client.0.smithi003.stdout:28.12% done 2023-10-18T03:56:21.511 INFO:tasks.workunit.client.0.smithi003.stdout:29.69% done 2023-10-18T03:56:21.511 INFO:tasks.workunit.client.0.smithi003.stdout:31.25% done 2023-10-18T03:56:21.511 INFO:tasks.workunit.client.0.smithi003.stdout:32.81% done 2023-10-18T03:56:21.511 INFO:tasks.workunit.client.0.smithi003.stdout:34.38% done 2023-10-18T03:56:21.512 INFO:tasks.workunit.client.0.smithi003.stdout:35.94% done 2023-10-18T03:56:21.512 INFO:tasks.workunit.client.0.smithi003.stdout:37.50% done 2023-10-18T03:56:21.512 INFO:tasks.workunit.client.0.smithi003.stdout:39.06% done 2023-10-18T03:56:21.512 INFO:tasks.workunit.client.0.smithi003.stdout:40.62% done 2023-10-18T03:56:21.512 INFO:tasks.workunit.client.0.smithi003.stdout:42.19% done 2023-10-18T03:56:21.512 INFO:tasks.workunit.client.0.smithi003.stdout:43.75% done 2023-10-18T03:56:21.513 INFO:tasks.workunit.client.0.smithi003.stdout:45.31% done 2023-10-18T03:56:21.513 INFO:tasks.workunit.client.0.smithi003.stdout:46.88% done 2023-10-18T03:56:21.513 INFO:tasks.workunit.client.0.smithi003.stdout:48.44% done 2023-10-18T03:56:21.513 INFO:tasks.workunit.client.0.smithi003.stdout:50.00% done 2023-10-18T03:56:21.513 INFO:tasks.workunit.client.0.smithi003.stdout:51.56% done 2023-10-18T03:56:21.514 INFO:tasks.workunit.client.0.smithi003.stdout:53.12% done 2023-10-18T03:56:21.514 INFO:tasks.workunit.client.0.smithi003.stdout:54.69% done 2023-10-18T03:56:21.514 INFO:tasks.workunit.client.0.smithi003.stdout:56.25% done 2023-10-18T03:56:21.514 INFO:tasks.workunit.client.0.smithi003.stdout:57.81% done 2023-10-18T03:56:21.514 INFO:tasks.workunit.client.0.smithi003.stdout:59.38% done 2023-10-18T03:56:21.514 INFO:tasks.workunit.client.0.smithi003.stdout:60.94% done 2023-10-18T03:56:21.515 INFO:tasks.workunit.client.0.smithi003.stdout:62.50% done 2023-10-18T03:56:21.515 INFO:tasks.workunit.client.0.smithi003.stdout:64.06% done 2023-10-18T03:56:21.515 INFO:tasks.workunit.client.0.smithi003.stdout:65.62% done 2023-10-18T03:56:21.515 INFO:tasks.workunit.client.0.smithi003.stdout:67.19% done 2023-10-18T03:56:21.515 INFO:tasks.workunit.client.0.smithi003.stdout:68.75% done 2023-10-18T03:56:21.516 INFO:tasks.workunit.client.0.smithi003.stdout:70.31% done 2023-10-18T03:56:21.516 INFO:tasks.workunit.client.0.smithi003.stdout:71.88% done 2023-10-18T03:56:21.516 INFO:tasks.workunit.client.0.smithi003.stdout:73.44% done 2023-10-18T03:56:21.516 INFO:tasks.workunit.client.0.smithi003.stdout:75.00% done 2023-10-18T03:56:21.516 INFO:tasks.workunit.client.0.smithi003.stdout:76.56% done 2023-10-18T03:56:21.516 INFO:tasks.workunit.client.0.smithi003.stdout:78.12% done 2023-10-18T03:56:21.517 INFO:tasks.workunit.client.0.smithi003.stdout:79.69% done 2023-10-18T03:56:21.517 INFO:tasks.workunit.client.0.smithi003.stdout:81.25% done 2023-10-18T03:56:21.517 INFO:tasks.workunit.client.0.smithi003.stdout:82.81% done 2023-10-18T03:56:21.517 INFO:tasks.workunit.client.0.smithi003.stdout:84.38% done 2023-10-18T03:56:21.517 INFO:tasks.workunit.client.0.smithi003.stdout:85.94% done 2023-10-18T03:56:21.517 INFO:tasks.workunit.client.0.smithi003.stdout:87.50% done 2023-10-18T03:56:21.518 INFO:tasks.workunit.client.0.smithi003.stdout:89.06% done 2023-10-18T03:56:21.518 INFO:tasks.workunit.client.0.smithi003.stdout:90.62% done 2023-10-18T03:56:21.518 INFO:tasks.workunit.client.0.smithi003.stdout:92.19% done 2023-10-18T03:56:21.518 INFO:tasks.workunit.client.0.smithi003.stdout:93.75% done 2023-10-18T03:56:21.518 INFO:tasks.workunit.client.0.smithi003.stdout:95.31% done 2023-10-18T03:56:21.518 INFO:tasks.workunit.client.0.smithi003.stdout:96.88% done 2023-10-18T03:56:21.519 INFO:tasks.workunit.client.0.smithi003.stdout:98.44% done 2023-10-18T03:56:21.519 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (486 ms) 2023-10-18T03:56:21.519 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2023-10-18T03:56:21.519 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:21.553 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:23.421 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestTrashPurge (1914 ms) 2023-10-18T03:56:23.421 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2023-10-18T03:56:23.422 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:23.756 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestTrashMoveAndRestore (335 ms) 2023-10-18T03:56:23.757 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestListWatchers 2023-10-18T03:56:23.757 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:23.963 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestListWatchers (206 ms) 2023-10-18T03:56:23.963 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2023-10-18T03:56:23.963 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:25.452 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestSetSnapById (1489 ms) 2023-10-18T03:56:25.452 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.Namespaces 2023-10-18T03:56:25.851 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.Namespaces (400 ms) 2023-10-18T03:56:25.852 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.NamespacesPP 2023-10-18T03:56:26.130 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:26.149 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:26.736 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.NamespacesPP (885 ms) 2023-10-18T03:56:26.736 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.Migration 2023-10-18T03:56:26.737 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:26.737 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:31.095 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.Migration (4358 ms) 2023-10-18T03:56:31.096 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.MigrationPP 2023-10-18T03:56:31.096 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:31.096 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:34.422 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.MigrationPP (3328 ms) 2023-10-18T03:56:34.423 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2023-10-18T03:56:34.423 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:34.539 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestGetAccessTimestamp (117 ms) 2023-10-18T03:56:34.539 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2023-10-18T03:56:34.539 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:34.697 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.TestGetModifyTimestamp (158 ms) 2023-10-18T03:56:34.697 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2023-10-18T03:56:34.698 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:36.224 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ZeroOverlapFlatten (1527 ms) 2023-10-18T03:56:36.225 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.PoolMetadata 2023-10-18T03:56:36.376 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.PoolMetadata (151 ms) 2023-10-18T03:56:36.376 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2023-10-18T03:56:36.489 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.PoolMetadataPP (114 ms) 2023-10-18T03:56:36.490 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.Config 2023-10-18T03:56:36.506 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:36.756 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.Config (266 ms) 2023-10-18T03:56:36.756 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ConfigPP 2023-10-18T03:56:36.769 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:37.267 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ConfigPP (511 ms) 2023-10-18T03:56:37.267 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2023-10-18T03:56:39.244 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:42.094 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:42.137 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:42.181 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:44.265 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.PoolStatsPP (6998 ms) 2023-10-18T03:56:44.265 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ImageSpec 2023-10-18T03:56:46.274 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:56:55.679 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.ImageSpec (11414 ms) 2023-10-18T03:56:55.679 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2023-10-18T03:56:57.040 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:57:09.593 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 3 2023-10-18T03:57:09.594 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 3 2023-10-18T03:57:09.594 INFO:tasks.workunit.client.0.smithi003.stdout:child: image213 2023-10-18T03:57:09.594 INFO:tasks.workunit.client.0.smithi003.stdout:child: image215 2023-10-18T03:57:09.594 INFO:tasks.workunit.client.0.smithi003.stdout:child: image214 2023-10-18T03:57:09.594 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-1/image213/105c66f6c439 2023-10-18T03:57:09.595 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:57:09.595 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-17/image214/105c408ac173 2023-10-18T03:57:09.595 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:57:09.595 INFO:tasks.workunit.client.0.smithi003.stdout:found child test-librbd-smithi003-85898-1/image215/105c559b37fe 2023-10-18T03:57:09.595 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:57:09.596 INFO:tasks.workunit.client.0.smithi003.stdout:num children is: 0 2023-10-18T03:57:09.596 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:57:09.596 INFO:tasks.workunit.client.0.smithi003.stdout:num snaps is: 0 2023-10-18T03:57:09.596 INFO:tasks.workunit.client.0.smithi003.stdout:expected: 0 2023-10-18T03:57:09.596 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.SnapRemoveWithChildMissing (13915 ms) 2023-10-18T03:57:09.597 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2023-10-18T03:57:11.102 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:57:24.407 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.QuiesceWatch (14814 ms) 2023-10-18T03:57:24.408 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2023-10-18T03:57:24.408 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:57:34.655 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (10247 ms) 2023-10-18T03:57:34.655 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2023-10-18T03:57:34.655 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_librbd.cc:12124: Skipped 2023-10-18T03:57:34.655 INFO:tasks.workunit.client.0.smithi003.stdout:Not supported by crimson yet. Skipped 2023-10-18T03:57:34.656 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError (0 ms) 2023-10-18T03:57:34.656 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2023-10-18T03:57:34.656 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:57:34.744 INFO:tasks.workunit.client.0.smithi003.stderr:test quiesce is not long enough to time out 2023-10-18T03:57:43.852 INFO:tasks.workunit.client.0.smithi003.stderr:test quiesce is timed out 2023-10-18T03:57:43.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 0 2023-10-18T03:57:44.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 1 2023-10-18T03:57:45.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 2 2023-10-18T03:57:46.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 3 2023-10-18T03:57:47.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 4 2023-10-18T03:57:48.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 5 2023-10-18T03:57:49.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 6 2023-10-18T03:57:50.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 7 2023-10-18T03:57:51.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 8 2023-10-18T03:57:52.956 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 9 2023-10-18T03:57:53.957 INFO:tasks.workunit.client.0.smithi003.stderr:waiting for timed out ... 10 2023-10-18T03:58:04.956 INFO:tasks.workunit.client.0.smithi003.stderr:test retry succeeds 2023-10-18T03:58:09.424 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (34770 ms) 2023-10-18T03:58:09.425 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.WriteZeroes 2023-10-18T03:58:09.425 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:58:09.680 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.WriteZeroes (256 ms) 2023-10-18T03:58:09.681 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2023-10-18T03:58:09.681 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:58:10.036 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (355 ms) 2023-10-18T03:58:10.036 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestLibRBD.ConcurrentOperations 2023-10-18T03:58:10.036 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_librbd.cc:12483: Skipped 2023-10-18T03:58:10.037 INFO:tasks.workunit.client.0.smithi003.stdout:Not supported by crimson yet. Skipped 2023-10-18T03:58:10.037 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations (0 ms) 2023-10-18T03:58:26.051 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 169 tests from TestLibRBD (803753 ms total) 2023-10-18T03:58:26.052 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:58:26.052 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T03:58:26.052 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 169 tests from 1 test suite ran. (819805 ms total) 2023-10-18T03:58:26.052 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 162 tests. 2023-10-18T03:58:26.052 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] 7 tests, listed below: 2023-10-18T03:58:26.052 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2023-10-18T03:58:26.053 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.ListChildrenTiered 2023-10-18T03:58:26.053 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2023-10-18T03:58:26.053 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2023-10-18T03:58:26.053 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.BreakLock 2023-10-18T03:58:26.053 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.QuiesceWatchError 2023-10-18T03:58:26.054 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestLibRBD.ConcurrentOperations 2023-10-18T03:58:26.054 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:58:26.054 INFO:tasks.workunit.client.0.smithi003.stdout: YOU HAVE 1 DISABLED TEST 2023-10-18T03:58:26.054 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:58:26.118 INFO:tasks.workunit.client.0.smithi003.stdout:seed 98984 2023-10-18T03:58:26.119 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = EncryptedFlattenTest/0.* 2023-10-18T03:58:26.119 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 7 tests from 1 test suite. 2023-10-18T03:58:26.119 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T03:58:26.119 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/0, where TypeParam = PlaintextUnderLUKS1 2023-10-18T03:58:26.179 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/0.Simple 2023-10-18T03:58:28.059 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:58:43.116 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/0.Simple (16936 ms) 2023-10-18T03:58:43.116 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/0.Grow 2023-10-18T03:58:43.116 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:58:53.656 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/0.Grow (10540 ms) 2023-10-18T03:58:53.657 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/0.Shrink 2023-10-18T03:58:53.657 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:59:04.083 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/0.Shrink (10427 ms) 2023-10-18T03:59:04.083 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/0.ShrinkToOne 2023-10-18T03:59:04.083 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:59:13.724 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/0.ShrinkToOne (9641 ms) 2023-10-18T03:59:13.724 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/0.ShrinkToOneAfterSnapshot 2023-10-18T03:59:13.725 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:59:25.648 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/0.ShrinkToOneAfterSnapshot (11925 ms) 2023-10-18T03:59:25.649 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/0.MinOverlap 2023-10-18T03:59:25.649 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:59:35.544 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/0.MinOverlap (9896 ms) 2023-10-18T03:59:35.545 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/0.ZeroOverlap 2023-10-18T03:59:35.545 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T03:59:46.495 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/0.ZeroOverlap (10951 ms) 2023-10-18T03:59:47.087 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/0 (80316 ms total) 2023-10-18T03:59:47.087 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T03:59:47.087 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T03:59:47.087 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 7 tests from 1 test suite ran. (80968 ms total) 2023-10-18T03:59:47.088 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 7 tests. 2023-10-18T03:59:47.145 INFO:tasks.workunit.client.0.smithi003.stdout:seed 99161 2023-10-18T03:59:47.146 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = EncryptedFlattenTest/1.* 2023-10-18T03:59:47.146 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 7 tests from 1 test suite. 2023-10-18T03:59:47.146 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T03:59:47.146 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/1, where TypeParam = PlaintextUnderLUKS2 2023-10-18T03:59:47.209 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/1.Simple 2023-10-18T03:59:49.094 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:00:05.669 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/1.Simple (18460 ms) 2023-10-18T04:00:05.669 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/1.Grow 2023-10-18T04:00:05.669 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:00:20.200 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/1.Grow (14532 ms) 2023-10-18T04:00:20.201 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/1.Shrink 2023-10-18T04:00:20.201 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:00:36.273 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/1.Shrink (16072 ms) 2023-10-18T04:00:36.273 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/1.ShrinkToOne 2023-10-18T04:00:36.274 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:00:50.786 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/1.ShrinkToOne (14514 ms) 2023-10-18T04:00:50.787 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/1.ShrinkToOneAfterSnapshot 2023-10-18T04:00:50.787 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:01:08.327 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/1.ShrinkToOneAfterSnapshot (17540 ms) 2023-10-18T04:01:08.327 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/1.MinOverlap 2023-10-18T04:01:08.327 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:01:20.524 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/1.MinOverlap (12198 ms) 2023-10-18T04:01:20.525 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/1.ZeroOverlap 2023-10-18T04:01:20.525 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:01:35.186 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/1.ZeroOverlap (14661 ms) 2023-10-18T04:01:36.131 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/1 (107977 ms total) 2023-10-18T04:01:36.131 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:01:36.132 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:01:36.132 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 7 tests from 1 test suite ran. (108985 ms total) 2023-10-18T04:01:36.132 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 7 tests. 2023-10-18T04:01:36.193 INFO:tasks.workunit.client.0.smithi003.stdout:seed 106789 2023-10-18T04:01:36.194 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = EncryptedFlattenTest/2.* 2023-10-18T04:01:36.194 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 7 tests from 1 test suite. 2023-10-18T04:01:36.194 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:01:36.194 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/2, where TypeParam = UnformattedLUKS1 2023-10-18T04:01:36.256 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/2.Simple 2023-10-18T04:01:38.102 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:01:55.104 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/2.Simple (18847 ms) 2023-10-18T04:01:55.104 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/2.Grow 2023-10-18T04:01:55.104 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:02:07.768 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/2.Grow (12664 ms) 2023-10-18T04:02:07.768 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/2.Shrink 2023-10-18T04:02:07.768 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:02:20.517 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/2.Shrink (12749 ms) 2023-10-18T04:02:20.517 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/2.ShrinkToOne 2023-10-18T04:02:20.518 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:02:32.886 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/2.ShrinkToOne (12370 ms) 2023-10-18T04:02:32.887 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/2.ShrinkToOneAfterSnapshot 2023-10-18T04:02:32.887 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:02:46.925 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/2.ShrinkToOneAfterSnapshot (14038 ms) 2023-10-18T04:02:46.925 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/2.MinOverlap 2023-10-18T04:02:46.925 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:02:59.903 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/2.MinOverlap (12979 ms) 2023-10-18T04:02:59.904 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/2.ZeroOverlap 2023-10-18T04:02:59.904 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:03:12.340 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/2.ZeroOverlap (12436 ms) 2023-10-18T04:03:13.148 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/2 (96083 ms total) 2023-10-18T04:03:13.149 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:03:13.149 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:03:13.149 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 7 tests from 1 test suite ran. (96955 ms total) 2023-10-18T04:03:13.149 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 7 tests. 2023-10-18T04:03:13.219 INFO:tasks.workunit.client.0.smithi003.stdout:seed 106858 2023-10-18T04:03:13.219 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = EncryptedFlattenTest/3.* 2023-10-18T04:03:13.219 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 7 tests from 1 test suite. 2023-10-18T04:03:13.220 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:03:13.220 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/3, where TypeParam = LUKS1UnderLUKS1 2023-10-18T04:03:13.282 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/3.Simple 2023-10-18T04:03:15.158 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:03:34.425 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/3.Simple (21144 ms) 2023-10-18T04:03:34.426 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/3.Grow 2023-10-18T04:03:34.426 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:03:51.238 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/3.Grow (16813 ms) 2023-10-18T04:03:51.239 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/3.Shrink 2023-10-18T04:03:51.239 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:04:05.353 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/3.Shrink (14115 ms) 2023-10-18T04:04:05.354 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/3.ShrinkToOne 2023-10-18T04:04:05.354 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:04:21.544 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/3.ShrinkToOne (16190 ms) 2023-10-18T04:04:21.544 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/3.ShrinkToOneAfterSnapshot 2023-10-18T04:04:21.545 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:04:38.996 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/3.ShrinkToOneAfterSnapshot (17453 ms) 2023-10-18T04:04:38.997 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/3.MinOverlap 2023-10-18T04:04:38.997 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:04:55.154 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/3.MinOverlap (16157 ms) 2023-10-18T04:04:55.155 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/3.ZeroOverlap 2023-10-18T04:04:55.155 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:05:11.113 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/3.ZeroOverlap (15960 ms) 2023-10-18T04:05:11.163 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/3 (117832 ms total) 2023-10-18T04:05:11.163 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:05:11.163 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:05:11.163 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 7 tests from 1 test suite ran. (117943 ms total) 2023-10-18T04:05:11.164 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 7 tests. 2023-10-18T04:05:11.215 INFO:tasks.workunit.client.0.smithi003.stdout:seed 106925 2023-10-18T04:05:11.216 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = EncryptedFlattenTest/4.* 2023-10-18T04:05:11.216 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 7 tests from 1 test suite. 2023-10-18T04:05:11.217 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:05:11.217 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/4, where TypeParam = LUKS1UnderLUKS2 2023-10-18T04:05:11.276 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/4.Simple 2023-10-18T04:05:13.154 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:05:38.038 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/4.Simple (26761 ms) 2023-10-18T04:05:38.038 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/4.Grow 2023-10-18T04:05:38.038 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:05:56.794 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/4.Grow (18756 ms) 2023-10-18T04:05:56.794 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/4.Shrink 2023-10-18T04:05:56.795 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:06:17.647 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/4.Shrink (20853 ms) 2023-10-18T04:06:17.647 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/4.ShrinkToOne 2023-10-18T04:06:17.647 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:06:41.318 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/4.ShrinkToOne (23671 ms) 2023-10-18T04:06:41.318 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/4.ShrinkToOneAfterSnapshot 2023-10-18T04:06:41.318 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:07:05.839 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/4.ShrinkToOneAfterSnapshot (24522 ms) 2023-10-18T04:07:05.839 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/4.MinOverlap 2023-10-18T04:07:05.840 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:07:27.459 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/4.MinOverlap (21619 ms) 2023-10-18T04:07:27.459 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/4.ZeroOverlap 2023-10-18T04:07:27.459 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:07:51.257 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/4.ZeroOverlap (23798 ms) 2023-10-18T04:07:52.222 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/4 (159980 ms total) 2023-10-18T04:07:52.222 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:07:52.222 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:07:52.223 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 7 tests from 1 test suite ran. (161007 ms total) 2023-10-18T04:07:52.223 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 7 tests. 2023-10-18T04:07:52.279 INFO:tasks.workunit.client.0.smithi003.stdout:seed 115609 2023-10-18T04:07:52.279 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = EncryptedFlattenTest/5.* 2023-10-18T04:07:52.279 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 7 tests from 1 test suite. 2023-10-18T04:07:52.279 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:07:52.280 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/5, where TypeParam = UnformattedLUKS2 2023-10-18T04:07:52.331 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/5.Simple 2023-10-18T04:07:54.233 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:08:15.925 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/5.Simple (23591 ms) 2023-10-18T04:08:15.926 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/5.Grow 2023-10-18T04:08:15.926 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:08:32.493 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/5.Grow (16570 ms) 2023-10-18T04:08:32.493 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/5.Shrink 2023-10-18T04:08:32.493 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:08:50.109 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/5.Shrink (17617 ms) 2023-10-18T04:08:50.110 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/5.ShrinkToOne 2023-10-18T04:08:50.110 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:09:06.542 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/5.ShrinkToOne (16432 ms) 2023-10-18T04:09:06.542 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/5.ShrinkToOneAfterSnapshot 2023-10-18T04:09:06.542 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:09:26.795 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/5.ShrinkToOneAfterSnapshot (20253 ms) 2023-10-18T04:09:26.795 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/5.MinOverlap 2023-10-18T04:09:26.795 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:09:43.672 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/5.MinOverlap (16878 ms) 2023-10-18T04:09:43.673 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/5.ZeroOverlap 2023-10-18T04:09:43.673 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:10:00.760 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/5.ZeroOverlap (17087 ms) 2023-10-18T04:10:00.903 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/5 (128428 ms total) 2023-10-18T04:10:00.903 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:10:00.903 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:10:00.903 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 7 tests from 1 test suite ran. (128623 ms total) 2023-10-18T04:10:00.904 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 7 tests. 2023-10-18T04:10:00.967 INFO:tasks.workunit.client.0.smithi003.stdout:seed 124471 2023-10-18T04:10:00.968 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = EncryptedFlattenTest/6.* 2023-10-18T04:10:00.968 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 7 tests from 1 test suite. 2023-10-18T04:10:00.968 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:10:00.969 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/6, where TypeParam = LUKS2UnderLUKS2 2023-10-18T04:10:01.028 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/6.Simple 2023-10-18T04:10:02.919 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:10:36.775 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/6.Simple (35746 ms) 2023-10-18T04:10:36.775 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/6.Grow 2023-10-18T04:10:36.776 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:11:02.419 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/6.Grow (25644 ms) 2023-10-18T04:11:02.419 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/6.Shrink 2023-10-18T04:11:02.420 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:11:25.354 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/6.Shrink (22936 ms) 2023-10-18T04:11:25.355 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/6.ShrinkToOne 2023-10-18T04:11:25.355 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:11:50.244 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/6.ShrinkToOne (24889 ms) 2023-10-18T04:11:50.244 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/6.ShrinkToOneAfterSnapshot 2023-10-18T04:11:50.244 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:12:13.430 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/6.ShrinkToOneAfterSnapshot (23187 ms) 2023-10-18T04:12:13.431 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/6.MinOverlap 2023-10-18T04:12:13.431 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:12:38.826 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/6.MinOverlap (25395 ms) 2023-10-18T04:12:38.826 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/6.ZeroOverlap 2023-10-18T04:12:38.827 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:13:01.176 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/6.ZeroOverlap (22350 ms) 2023-10-18T04:13:01.952 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/6 (180147 ms total) 2023-10-18T04:13:01.953 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:13:01.953 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:13:01.953 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 7 tests from 1 test suite ran. (180985 ms total) 2023-10-18T04:13:01.953 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 7 tests. 2023-10-18T04:13:02.025 INFO:tasks.workunit.client.0.smithi003.stdout:seed 138621 2023-10-18T04:13:02.025 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = EncryptedFlattenTest/7.* 2023-10-18T04:13:02.026 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 7 tests from 1 test suite. 2023-10-18T04:13:02.026 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:13:02.026 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/7, where TypeParam = LUKS2UnderLUKS1 2023-10-18T04:13:02.080 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/7.Simple 2023-10-18T04:13:03.316 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:13:35.233 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/7.Simple (33152 ms) 2023-10-18T04:13:35.234 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/7.Grow 2023-10-18T04:13:35.234 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:13:58.152 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/7.Grow (22919 ms) 2023-10-18T04:13:58.153 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/7.Shrink 2023-10-18T04:13:58.153 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:14:23.476 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/7.Shrink (25323 ms) 2023-10-18T04:14:23.476 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/7.ShrinkToOne 2023-10-18T04:14:23.476 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:14:48.386 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/7.ShrinkToOne (24910 ms) 2023-10-18T04:14:48.386 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/7.ShrinkToOneAfterSnapshot 2023-10-18T04:14:48.386 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:15:16.679 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/7.ShrinkToOneAfterSnapshot (28293 ms) 2023-10-18T04:15:16.679 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/7.MinOverlap 2023-10-18T04:15:16.679 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:15:41.720 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/7.MinOverlap (25041 ms) 2023-10-18T04:15:41.721 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] EncryptedFlattenTest/7.ZeroOverlap 2023-10-18T04:15:41.721 INFO:tasks.workunit.client.0.smithi003.stdout:using new format! 2023-10-18T04:16:06.663 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] EncryptedFlattenTest/7.ZeroOverlap (24943 ms) 2023-10-18T04:16:06.990 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 7 tests from EncryptedFlattenTest/7 (184582 ms total) 2023-10-18T04:16:06.990 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:16:06.990 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:16:06.991 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 7 tests from 1 test suite ran. (184964 ms total) 2023-10-18T04:16:06.991 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 7 tests. 2023-10-18T04:16:07.053 INFO:tasks.workunit.client.0.smithi003.stdout:seed 146297 2023-10-18T04:16:07.053 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = TestImageWatcher.* 2023-10-18T04:16:07.053 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 17 tests from 1 test suite. 2023-10-18T04:16:07.054 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:16:07.054 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 17 tests from TestImageWatcher 2023-10-18T04:16:08.371 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyHeaderUpdate 2023-10-18T04:16:13.625 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyHeaderUpdate (5254 ms) 2023-10-18T04:16:13.625 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyFlatten 2023-10-18T04:16:13.762 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyFlatten (138 ms) 2023-10-18T04:16:13.763 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyResize 2023-10-18T04:16:13.906 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyResize (143 ms) 2023-10-18T04:16:13.906 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyRebuildObjectMap 2023-10-18T04:16:14.041 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyRebuildObjectMap (136 ms) 2023-10-18T04:16:14.042 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyUpdateFeatures 2023-10-18T04:16:14.173 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyUpdateFeatures (131 ms) 2023-10-18T04:16:14.173 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifySnapCreate 2023-10-18T04:16:14.311 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifySnapCreate (139 ms) 2023-10-18T04:16:14.312 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifySnapCreateError 2023-10-18T04:16:14.446 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifySnapCreateError (134 ms) 2023-10-18T04:16:14.446 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifySnapRename 2023-10-18T04:16:14.579 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifySnapRename (133 ms) 2023-10-18T04:16:14.579 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifySnapRenameError 2023-10-18T04:16:14.709 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifySnapRenameError (131 ms) 2023-10-18T04:16:14.709 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifySnapRemove 2023-10-18T04:16:14.842 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifySnapRemove (133 ms) 2023-10-18T04:16:14.842 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifySnapProtect 2023-10-18T04:16:14.975 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifySnapProtect (133 ms) 2023-10-18T04:16:14.976 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifySnapUnprotect 2023-10-18T04:16:15.117 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifySnapUnprotect (141 ms) 2023-10-18T04:16:15.117 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyRename 2023-10-18T04:16:15.251 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyRename (135 ms) 2023-10-18T04:16:15.252 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyAsyncTimedOut 2023-10-18T04:16:15.380 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyAsyncTimedOut (129 ms) 2023-10-18T04:16:15.381 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyAsyncError 2023-10-18T04:16:15.515 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyAsyncError (134 ms) 2023-10-18T04:16:15.515 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyAsyncCompleteError 2023-10-18T04:16:15.650 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyAsyncCompleteError (135 ms) 2023-10-18T04:16:15.650 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestImageWatcher.NotifyAsyncRequestTimedOut 2023-10-18T04:16:15.785 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestImageWatcher.NotifyAsyncRequestTimedOut (136 ms) 2023-10-18T04:16:15.987 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 17 tests from TestImageWatcher (7415 ms total) 2023-10-18T04:16:15.988 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:16:15.988 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:16:15.988 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 17 tests from 1 test suite ran. (8934 ms total) 2023-10-18T04:16:15.988 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 17 tests. 2023-10-18T04:16:16.045 INFO:tasks.workunit.client.0.smithi003.stdout:seed 146363 2023-10-18T04:16:16.045 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = TestInternal.* 2023-10-18T04:16:16.046 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 34 tests from 1 test suite. 2023-10-18T04:16:16.046 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:16:16.046 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 34 tests from TestInternal 2023-10-18T04:16:17.996 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.OpenByID 2023-10-18T04:16:28.577 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.OpenByID (10581 ms) 2023-10-18T04:16:28.577 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.OpenSnapDNE 2023-10-18T04:16:28.714 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.OpenSnapDNE (137 ms) 2023-10-18T04:16:28.714 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.IsExclusiveLockOwner 2023-10-18T04:16:28.868 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.IsExclusiveLockOwner (154 ms) 2023-10-18T04:16:28.868 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.ResizeLocksImage 2023-10-18T04:16:29.033 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.ResizeLocksImage (166 ms) 2023-10-18T04:16:29.033 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.ResizeFailsToLockImage 2023-10-18T04:16:29.153 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.ResizeFailsToLockImage (120 ms) 2023-10-18T04:16:29.153 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SnapCreateLocksImage 2023-10-18T04:16:31.543 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.SnapCreateLocksImage (2389 ms) 2023-10-18T04:16:31.543 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SnapCreateFailsToLockImage 2023-10-18T04:16:31.654 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.SnapCreateFailsToLockImage (111 ms) 2023-10-18T04:16:31.654 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SnapRollbackLocksImage 2023-10-18T04:16:35.543 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.SnapRollbackLocksImage (3890 ms) 2023-10-18T04:16:35.544 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SnapRollbackFailsToLockImage 2023-10-18T04:16:38.634 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.SnapRollbackFailsToLockImage (3090 ms) 2023-10-18T04:16:38.634 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SnapSetReleasesLock 2023-10-18T04:16:41.694 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.SnapSetReleasesLock (3060 ms) 2023-10-18T04:16:41.694 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.FlattenLocksImage 2023-10-18T04:16:45.559 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.FlattenLocksImage (3866 ms) 2023-10-18T04:16:45.560 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.FlattenFailsToLockImage 2023-10-18T04:16:50.297 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.FlattenFailsToLockImage (4738 ms) 2023-10-18T04:16:50.298 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklisted 2023-10-18T04:16:50.611 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_internal.cc:382: Skipped 2023-10-18T04:16:50.611 INFO:tasks.workunit.client.0.smithi003.stdout:Not supported by crimson yet. Skipped 2023-10-18T04:16:50.611 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.WriteFailsToLockImageBlocklisted (313 ms) 2023-10-18T04:16:50.611 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklistedWatch 2023-10-18T04:16:50.655 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_internal.cc:416: Skipped 2023-10-18T04:16:50.655 INFO:tasks.workunit.client.0.smithi003.stdout:Not supported by crimson yet. Skipped 2023-10-18T04:16:50.655 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.WriteFailsToLockImageBlocklistedWatch (44 ms) 2023-10-18T04:16:50.655 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.AioWriteRequestsLock 2023-10-18T04:16:50.835 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.AioWriteRequestsLock (180 ms) 2023-10-18T04:16:50.835 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.AioDiscardRequestsLock 2023-10-18T04:16:50.996 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.AioDiscardRequestsLock (162 ms) 2023-10-18T04:16:50.997 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.CancelAsyncResize 2023-10-18T04:16:51.326 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.CancelAsyncResize (330 ms) 2023-10-18T04:16:51.327 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.MultipleResize 2023-10-18T04:16:51.769 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.MultipleResize (442 ms) 2023-10-18T04:16:51.769 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.Metadata 2023-10-18T04:16:52.138 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.Metadata (369 ms) 2023-10-18T04:16:52.138 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.MetadataConfApply 2023-10-18T04:16:52.359 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.MetadataConfApply (221 ms) 2023-10-18T04:16:52.359 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SnapshotCopyup 2023-10-18T04:16:53.442 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_internal.cc:662: Skipped 2023-10-18T04:16:53.442 INFO:tasks.workunit.client.0.smithi003.stdout:Not supported by crimson yet. Skipped 2023-10-18T04:16:53.442 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.SnapshotCopyup (1083 ms) 2023-10-18T04:16:53.443 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SnapshotCopyupZeros 2023-10-18T04:16:57.158 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.SnapshotCopyupZeros (3716 ms) 2023-10-18T04:16:57.158 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SnapshotCopyupZerosMigration 2023-10-18T04:16:59.569 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.SnapshotCopyupZerosMigration (2411 ms) 2023-10-18T04:16:59.569 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.ResizeCopyup 2023-10-18T04:17:03.767 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.ResizeCopyup (4199 ms) 2023-10-18T04:17:03.768 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.DiscardCopyup 2023-10-18T04:17:03.810 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_internal.cc:1056: Skipped 2023-10-18T04:17:03.811 INFO:tasks.workunit.client.0.smithi003.stdout:Skipping due to unmet REQUIRE 2023-10-18T04:17:03.811 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.DiscardCopyup (43 ms) 2023-10-18T04:17:03.811 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.ImageOptions 2023-10-18T04:17:03.853 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.ImageOptions (43 ms) 2023-10-18T04:17:03.853 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.WriteFullCopyup 2023-10-18T04:17:12.126 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.WriteFullCopyup (8273 ms) 2023-10-18T04:17:12.127 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.DiffIterateCloneOverwrite 2023-10-18T04:17:16.619 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.DiffIterateCloneOverwrite (4493 ms) 2023-10-18T04:17:16.620 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.TestCoR 2023-10-18T04:17:16.665 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_internal.cc:1313: Skipped 2023-10-18T04:17:16.665 INFO:tasks.workunit.client.0.smithi003.stdout:Skipping due to disabled copy-on-read 2023-10-18T04:17:16.665 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.TestCoR (45 ms) 2023-10-18T04:17:16.665 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.FlattenNoEmptyObjects 2023-10-18T04:17:18.356 INFO:tasks.workunit.client.0.smithi003.stdout:made parent image "image39": 4194304K (1024 * 4K) 2023-10-18T04:17:18.356 INFO:tasks.workunit.client.0.smithi003.stdout:generated random write map: 2023-10-18T04:17:18.356 INFO:tasks.workunit.client.0.smithi003.stdout: [11 , 46413 ] 2023-10-18T04:17:18.356 INFO:tasks.workunit.client.0.smithi003.stdout: [23 , 95986 ] 2023-10-18T04:17:18.357 INFO:tasks.workunit.client.0.smithi003.stdout: [30 , 124437 ] 2023-10-18T04:17:18.357 INFO:tasks.workunit.client.0.smithi003.stdout: [32 , 132797 ] 2023-10-18T04:17:18.357 INFO:tasks.workunit.client.0.smithi003.stdout: [57 , 234669 ] 2023-10-18T04:17:18.357 INFO:tasks.workunit.client.0.smithi003.stdout: [65 , 267707 ] 2023-10-18T04:17:18.357 INFO:tasks.workunit.client.0.smithi003.stdout: [82 , 336062 ] 2023-10-18T04:17:18.358 INFO:tasks.workunit.client.0.smithi003.stdout: [88 , 362302 ] 2023-10-18T04:17:18.358 INFO:tasks.workunit.client.0.smithi003.stdout: [102 , 420591 ] 2023-10-18T04:17:18.358 INFO:tasks.workunit.client.0.smithi003.stdout: [108 , 445444 ] 2023-10-18T04:17:18.358 INFO:tasks.workunit.client.0.smithi003.stdout: [147 , 604784 ] 2023-10-18T04:17:18.358 INFO:tasks.workunit.client.0.smithi003.stdout: [165 , 677078 ] 2023-10-18T04:17:18.359 INFO:tasks.workunit.client.0.smithi003.stdout: [167 , 685958 ] 2023-10-18T04:17:18.359 INFO:tasks.workunit.client.0.smithi003.stdout: [169 , 692770 ] 2023-10-18T04:17:18.359 INFO:tasks.workunit.client.0.smithi003.stdout: [175 , 719666 ] 2023-10-18T04:17:18.359 INFO:tasks.workunit.client.0.smithi003.stdout: [178 , 729121 ] 2023-10-18T04:17:18.359 INFO:tasks.workunit.client.0.smithi003.stdout: [186 , 765065 ] 2023-10-18T04:17:18.359 INFO:tasks.workunit.client.0.smithi003.stdout: [214 , 878248 ] 2023-10-18T04:17:18.360 INFO:tasks.workunit.client.0.smithi003.stdout: [236 , 966982 ] 2023-10-18T04:17:18.361 INFO:tasks.workunit.client.0.smithi003.stdout: [248 , 1017353 ] 2023-10-18T04:17:18.361 INFO:tasks.workunit.client.0.smithi003.stdout: [278 , 1138915 ] 2023-10-18T04:17:18.361 INFO:tasks.workunit.client.0.smithi003.stdout: [284 , 1163936 ] 2023-10-18T04:17:18.361 INFO:tasks.workunit.client.0.smithi003.stdout: [293 , 1200265 ] 2023-10-18T04:17:18.362 INFO:tasks.workunit.client.0.smithi003.stdout: [297 , 1220061 ] 2023-10-18T04:17:18.362 INFO:tasks.workunit.client.0.smithi003.stdout: [301 , 1235159 ] 2023-10-18T04:17:18.362 INFO:tasks.workunit.client.0.smithi003.stdout: [307 , 1259218 ] 2023-10-18T04:17:18.362 INFO:tasks.workunit.client.0.smithi003.stdout: [310 , 1272115 ] 2023-10-18T04:17:18.362 INFO:tasks.workunit.client.0.smithi003.stdout: [312 , 1281143 ] 2023-10-18T04:17:18.363 INFO:tasks.workunit.client.0.smithi003.stdout: [325 , 1333004 ] 2023-10-18T04:17:18.363 INFO:tasks.workunit.client.0.smithi003.stdout: [338 , 1384849 ] 2023-10-18T04:17:18.363 INFO:tasks.workunit.client.0.smithi003.stdout: [365 , 1496651 ] 2023-10-18T04:17:18.363 INFO:tasks.workunit.client.0.smithi003.stdout: [369 , 1514184 ] 2023-10-18T04:17:18.363 INFO:tasks.workunit.client.0.smithi003.stdout: [374 , 1532405 ] 2023-10-18T04:17:18.364 INFO:tasks.workunit.client.0.smithi003.stdout: [388 , 1589924 ] 2023-10-18T04:17:18.364 INFO:tasks.workunit.client.0.smithi003.stdout: [391 , 1602074 ] 2023-10-18T04:17:18.364 INFO:tasks.workunit.client.0.smithi003.stdout: [392 , 1609103 ] 2023-10-18T04:17:18.364 INFO:tasks.workunit.client.0.smithi003.stdout: [396 , 1625464 ] 2023-10-18T04:17:18.364 INFO:tasks.workunit.client.0.smithi003.stdout: [402 , 1648920 ] 2023-10-18T04:17:18.365 INFO:tasks.workunit.client.0.smithi003.stdout: [423 , 1734951 ] 2023-10-18T04:17:18.365 INFO:tasks.workunit.client.0.smithi003.stdout: [426 , 1746741 ] 2023-10-18T04:17:18.365 INFO:tasks.workunit.client.0.smithi003.stdout: [470 , 1928208 ] 2023-10-18T04:17:18.365 INFO:tasks.workunit.client.0.smithi003.stdout: [479 , 1962482 ] 2023-10-18T04:17:18.365 INFO:tasks.workunit.client.0.smithi003.stdout: [481 , 1973172 ] 2023-10-18T04:17:18.366 INFO:tasks.workunit.client.0.smithi003.stdout: [494 , 2026673 ] 2023-10-18T04:17:18.366 INFO:tasks.workunit.client.0.smithi003.stdout: [498 , 2040029 ] 2023-10-18T04:17:18.366 INFO:tasks.workunit.client.0.smithi003.stdout: [521 , 2136347 ] 2023-10-18T04:17:18.366 INFO:tasks.workunit.client.0.smithi003.stdout: [525 , 2152371 ] 2023-10-18T04:17:18.366 INFO:tasks.workunit.client.0.smithi003.stdout: [531 , 2176384 ] 2023-10-18T04:17:18.367 INFO:tasks.workunit.client.0.smithi003.stdout: [551 , 2259716 ] 2023-10-18T04:17:18.367 INFO:tasks.workunit.client.0.smithi003.stdout: [557 , 2282317 ] 2023-10-18T04:17:18.367 INFO:tasks.workunit.client.0.smithi003.stdout: [607 , 2486982 ] 2023-10-18T04:17:18.367 INFO:tasks.workunit.client.0.smithi003.stdout: [609 , 2495257 ] 2023-10-18T04:17:18.367 INFO:tasks.workunit.client.0.smithi003.stdout: [621 , 2544353 ] 2023-10-18T04:17:18.367 INFO:tasks.workunit.client.0.smithi003.stdout: [625 , 2562204 ] 2023-10-18T04:17:18.368 INFO:tasks.workunit.client.0.smithi003.stdout: [640 , 2624010 ] 2023-10-18T04:17:18.368 INFO:tasks.workunit.client.0.smithi003.stdout: [643 , 2637174 ] 2023-10-18T04:17:18.368 INFO:tasks.workunit.client.0.smithi003.stdout: [672 , 2755447 ] 2023-10-18T04:17:18.368 INFO:tasks.workunit.client.0.smithi003.stdout: [675 , 2765100 ] 2023-10-18T04:17:18.368 INFO:tasks.workunit.client.0.smithi003.stdout: [680 , 2788571 ] 2023-10-18T04:17:18.369 INFO:tasks.workunit.client.0.smithi003.stdout: [683 , 2798383 ] 2023-10-18T04:17:18.369 INFO:tasks.workunit.client.0.smithi003.stdout: [689 , 2824964 ] 2023-10-18T04:17:18.369 INFO:tasks.workunit.client.0.smithi003.stdout: [692 , 2836446 ] 2023-10-18T04:17:18.369 INFO:tasks.workunit.client.0.smithi003.stdout: [703 , 2882273 ] 2023-10-18T04:17:18.369 INFO:tasks.workunit.client.0.smithi003.stdout: [706 , 2892241 ] 2023-10-18T04:17:18.369 INFO:tasks.workunit.client.0.smithi003.stdout: [711 , 2912304 ] 2023-10-18T04:17:18.370 INFO:tasks.workunit.client.0.smithi003.stdout: [735 , 3011524 ] 2023-10-18T04:17:18.370 INFO:tasks.workunit.client.0.smithi003.stdout: [750 , 3072327 ] 2023-10-18T04:17:18.370 INFO:tasks.workunit.client.0.smithi003.stdout: [752 , 3081491 ] 2023-10-18T04:17:18.370 INFO:tasks.workunit.client.0.smithi003.stdout: [756 , 3097368 ] 2023-10-18T04:17:18.370 INFO:tasks.workunit.client.0.smithi003.stdout: [757 , 3100837 ] 2023-10-18T04:17:18.371 INFO:tasks.workunit.client.0.smithi003.stdout: [761 , 3118896 ] 2023-10-18T04:17:18.371 INFO:tasks.workunit.client.0.smithi003.stdout: [766 , 3138997 ] 2023-10-18T04:17:18.371 INFO:tasks.workunit.client.0.smithi003.stdout: [778 , 3187312 ] 2023-10-18T04:17:18.371 INFO:tasks.workunit.client.0.smithi003.stdout: [798 , 3271466 ] 2023-10-18T04:17:18.371 INFO:tasks.workunit.client.0.smithi003.stdout: [812 , 3328094 ] 2023-10-18T04:17:18.371 INFO:tasks.workunit.client.0.smithi003.stdout: [818 , 3350586 ] 2023-10-18T04:17:18.372 INFO:tasks.workunit.client.0.smithi003.stdout: [822 , 3369226 ] 2023-10-18T04:17:18.372 INFO:tasks.workunit.client.0.smithi003.stdout: [849 , 3479601 ] 2023-10-18T04:17:18.372 INFO:tasks.workunit.client.0.smithi003.stdout: [880 , 3606621 ] 2023-10-18T04:17:18.372 INFO:tasks.workunit.client.0.smithi003.stdout: [881 , 3611726 ] 2023-10-18T04:17:18.372 INFO:tasks.workunit.client.0.smithi003.stdout: [888 , 3637701 ] 2023-10-18T04:17:18.373 INFO:tasks.workunit.client.0.smithi003.stdout: [890 , 3645939 ] 2023-10-18T04:17:18.373 INFO:tasks.workunit.client.0.smithi003.stdout: [905 , 3708736 ] 2023-10-18T04:17:18.373 INFO:tasks.workunit.client.0.smithi003.stdout: [913 , 3742042 ] 2023-10-18T04:17:18.373 INFO:tasks.workunit.client.0.smithi003.stdout: [950 , 3891387 ] 2023-10-18T04:17:18.373 INFO:tasks.workunit.client.0.smithi003.stdout: [952 , 3899903 ] 2023-10-18T04:17:18.373 INFO:tasks.workunit.client.0.smithi003.stdout: [957 , 3923233 ] 2023-10-18T04:17:18.374 INFO:tasks.workunit.client.0.smithi003.stdout: [965 , 3955609 ] 2023-10-18T04:17:18.374 INFO:tasks.workunit.client.0.smithi003.stdout: [966 , 3957430 ] 2023-10-18T04:17:18.374 INFO:tasks.workunit.client.0.smithi003.stdout: [972 , 3982101 ] 2023-10-18T04:17:18.374 INFO:tasks.workunit.client.0.smithi003.stdout: [975 , 3996813 ] 2023-10-18T04:17:18.374 INFO:tasks.workunit.client.0.smithi003.stdout: [982 , 4023385 ] 2023-10-18T04:17:18.375 INFO:tasks.workunit.client.0.smithi003.stdout: [997 , 4086961 ] 2023-10-18T04:17:18.375 INFO:tasks.workunit.client.0.smithi003.stdout: [1004 , 4114710 ] 2023-10-18T04:17:18.375 INFO:tasks.workunit.client.0.smithi003.stdout: [1019 , 4174825 ] 2023-10-18T04:17:18.375 INFO:tasks.workunit.client.0.smithi003.stdout: [1021 , 4184853 ] 2023-10-18T04:17:18.375 INFO:tasks.workunit.client.0.smithi003.stdout:write data based on random map 2023-10-18T04:17:18.376 INFO:tasks.workunit.client.0.smithi003.stdout: write object-11 2023-10-18T04:17:18.376 INFO:tasks.workunit.client.0.smithi003.stdout: write object-23 2023-10-18T04:17:18.376 INFO:tasks.workunit.client.0.smithi003.stdout: write object-30 2023-10-18T04:17:18.376 INFO:tasks.workunit.client.0.smithi003.stdout: write object-32 2023-10-18T04:17:18.376 INFO:tasks.workunit.client.0.smithi003.stdout: write object-57 2023-10-18T04:17:18.376 INFO:tasks.workunit.client.0.smithi003.stdout: write object-65 2023-10-18T04:17:18.377 INFO:tasks.workunit.client.0.smithi003.stdout: write object-82 2023-10-18T04:17:18.377 INFO:tasks.workunit.client.0.smithi003.stdout: write object-88 2023-10-18T04:17:18.377 INFO:tasks.workunit.client.0.smithi003.stdout: write object-102 2023-10-18T04:17:18.377 INFO:tasks.workunit.client.0.smithi003.stdout: write object-108 2023-10-18T04:17:18.377 INFO:tasks.workunit.client.0.smithi003.stdout: write object-147 2023-10-18T04:17:18.378 INFO:tasks.workunit.client.0.smithi003.stdout: write object-165 2023-10-18T04:17:18.378 INFO:tasks.workunit.client.0.smithi003.stdout: write object-167 2023-10-18T04:17:18.378 INFO:tasks.workunit.client.0.smithi003.stdout: write object-169 2023-10-18T04:17:18.378 INFO:tasks.workunit.client.0.smithi003.stdout: write object-175 2023-10-18T04:17:18.378 INFO:tasks.workunit.client.0.smithi003.stdout: write object-178 2023-10-18T04:17:18.379 INFO:tasks.workunit.client.0.smithi003.stdout: write object-186 2023-10-18T04:17:18.379 INFO:tasks.workunit.client.0.smithi003.stdout: write object-214 2023-10-18T04:17:18.379 INFO:tasks.workunit.client.0.smithi003.stdout: write object-236 2023-10-18T04:17:18.379 INFO:tasks.workunit.client.0.smithi003.stdout: write object-248 2023-10-18T04:17:18.379 INFO:tasks.workunit.client.0.smithi003.stdout: write object-278 2023-10-18T04:17:18.379 INFO:tasks.workunit.client.0.smithi003.stdout: write object-284 2023-10-18T04:17:18.380 INFO:tasks.workunit.client.0.smithi003.stdout: write object-293 2023-10-18T04:17:18.380 INFO:tasks.workunit.client.0.smithi003.stdout: write object-297 2023-10-18T04:17:18.380 INFO:tasks.workunit.client.0.smithi003.stdout: write object-301 2023-10-18T04:17:18.380 INFO:tasks.workunit.client.0.smithi003.stdout: write object-307 2023-10-18T04:17:18.380 INFO:tasks.workunit.client.0.smithi003.stdout: write object-310 2023-10-18T04:17:18.380 INFO:tasks.workunit.client.0.smithi003.stdout: write object-312 2023-10-18T04:17:18.381 INFO:tasks.workunit.client.0.smithi003.stdout: write object-325 2023-10-18T04:17:18.381 INFO:tasks.workunit.client.0.smithi003.stdout: write object-338 2023-10-18T04:17:18.381 INFO:tasks.workunit.client.0.smithi003.stdout: write object-365 2023-10-18T04:17:18.381 INFO:tasks.workunit.client.0.smithi003.stdout: write object-369 2023-10-18T04:17:18.381 INFO:tasks.workunit.client.0.smithi003.stdout: write object-374 2023-10-18T04:17:18.382 INFO:tasks.workunit.client.0.smithi003.stdout: write object-388 2023-10-18T04:17:18.382 INFO:tasks.workunit.client.0.smithi003.stdout: write object-391 2023-10-18T04:17:18.382 INFO:tasks.workunit.client.0.smithi003.stdout: write object-392 2023-10-18T04:17:18.382 INFO:tasks.workunit.client.0.smithi003.stdout: write object-396 2023-10-18T04:17:18.382 INFO:tasks.workunit.client.0.smithi003.stdout: write object-402 2023-10-18T04:17:18.383 INFO:tasks.workunit.client.0.smithi003.stdout: write object-423 2023-10-18T04:17:18.383 INFO:tasks.workunit.client.0.smithi003.stdout: write object-426 2023-10-18T04:17:18.383 INFO:tasks.workunit.client.0.smithi003.stdout: write object-470 2023-10-18T04:17:18.383 INFO:tasks.workunit.client.0.smithi003.stdout: write object-479 2023-10-18T04:17:18.383 INFO:tasks.workunit.client.0.smithi003.stdout: write object-481 2023-10-18T04:17:18.383 INFO:tasks.workunit.client.0.smithi003.stdout: write object-494 2023-10-18T04:17:18.384 INFO:tasks.workunit.client.0.smithi003.stdout: write object-498 2023-10-18T04:17:18.384 INFO:tasks.workunit.client.0.smithi003.stdout: write object-521 2023-10-18T04:17:18.384 INFO:tasks.workunit.client.0.smithi003.stdout: write object-525 2023-10-18T04:17:18.384 INFO:tasks.workunit.client.0.smithi003.stdout: write object-531 2023-10-18T04:17:18.384 INFO:tasks.workunit.client.0.smithi003.stdout: write object-551 2023-10-18T04:17:18.384 INFO:tasks.workunit.client.0.smithi003.stdout: write object-557 2023-10-18T04:17:18.385 INFO:tasks.workunit.client.0.smithi003.stdout: write object-607 2023-10-18T04:17:18.385 INFO:tasks.workunit.client.0.smithi003.stdout: write object-609 2023-10-18T04:17:18.385 INFO:tasks.workunit.client.0.smithi003.stdout: write object-621 2023-10-18T04:17:18.385 INFO:tasks.workunit.client.0.smithi003.stdout: write object-625 2023-10-18T04:17:18.385 INFO:tasks.workunit.client.0.smithi003.stdout: write object-640 2023-10-18T04:17:18.386 INFO:tasks.workunit.client.0.smithi003.stdout: write object-643 2023-10-18T04:17:18.386 INFO:tasks.workunit.client.0.smithi003.stdout: write object-672 2023-10-18T04:17:18.386 INFO:tasks.workunit.client.0.smithi003.stdout: write object-675 2023-10-18T04:17:18.386 INFO:tasks.workunit.client.0.smithi003.stdout: write object-680 2023-10-18T04:17:18.386 INFO:tasks.workunit.client.0.smithi003.stdout: write object-683 2023-10-18T04:17:18.387 INFO:tasks.workunit.client.0.smithi003.stdout: write object-689 2023-10-18T04:17:18.387 INFO:tasks.workunit.client.0.smithi003.stdout: write object-692 2023-10-18T04:17:18.387 INFO:tasks.workunit.client.0.smithi003.stdout: write object-703 2023-10-18T04:17:18.387 INFO:tasks.workunit.client.0.smithi003.stdout: write object-706 2023-10-18T04:17:18.387 INFO:tasks.workunit.client.0.smithi003.stdout: write object-711 2023-10-18T04:17:18.388 INFO:tasks.workunit.client.0.smithi003.stdout: write object-735 2023-10-18T04:17:18.388 INFO:tasks.workunit.client.0.smithi003.stdout: write object-750 2023-10-18T04:17:18.388 INFO:tasks.workunit.client.0.smithi003.stdout: write object-752 2023-10-18T04:17:18.388 INFO:tasks.workunit.client.0.smithi003.stdout: write object-756 2023-10-18T04:17:18.388 INFO:tasks.workunit.client.0.smithi003.stdout: write object-757 2023-10-18T04:17:18.388 INFO:tasks.workunit.client.0.smithi003.stdout: write object-761 2023-10-18T04:17:18.389 INFO:tasks.workunit.client.0.smithi003.stdout: write object-766 2023-10-18T04:17:18.389 INFO:tasks.workunit.client.0.smithi003.stdout: write object-778 2023-10-18T04:17:18.389 INFO:tasks.workunit.client.0.smithi003.stdout: write object-798 2023-10-18T04:17:18.389 INFO:tasks.workunit.client.0.smithi003.stdout: write object-812 2023-10-18T04:17:18.389 INFO:tasks.workunit.client.0.smithi003.stdout: write object-818 2023-10-18T04:17:18.389 INFO:tasks.workunit.client.0.smithi003.stdout: write object-822 2023-10-18T04:17:18.390 INFO:tasks.workunit.client.0.smithi003.stdout: write object-849 2023-10-18T04:17:18.390 INFO:tasks.workunit.client.0.smithi003.stdout: write object-880 2023-10-18T04:17:18.390 INFO:tasks.workunit.client.0.smithi003.stdout: write object-881 2023-10-18T04:17:18.390 INFO:tasks.workunit.client.0.smithi003.stdout: write object-888 2023-10-18T04:17:18.390 INFO:tasks.workunit.client.0.smithi003.stdout: write object-890 2023-10-18T04:17:18.391 INFO:tasks.workunit.client.0.smithi003.stdout: write object-905 2023-10-18T04:17:18.391 INFO:tasks.workunit.client.0.smithi003.stdout: write object-913 2023-10-18T04:17:18.391 INFO:tasks.workunit.client.0.smithi003.stdout: write object-950 2023-10-18T04:17:18.391 INFO:tasks.workunit.client.0.smithi003.stdout: write object-952 2023-10-18T04:17:18.391 INFO:tasks.workunit.client.0.smithi003.stdout: write object-957 2023-10-18T04:17:18.392 INFO:tasks.workunit.client.0.smithi003.stdout: write object-965 2023-10-18T04:17:22.367 INFO:tasks.workunit.client.0.smithi003.stdout: write object-966 2023-10-18T04:17:22.367 INFO:tasks.workunit.client.0.smithi003.stdout: write object-972 2023-10-18T04:17:22.367 INFO:tasks.workunit.client.0.smithi003.stdout: write object-975 2023-10-18T04:17:22.368 INFO:tasks.workunit.client.0.smithi003.stdout: write object-982 2023-10-18T04:17:22.368 INFO:tasks.workunit.client.0.smithi003.stdout: write object-997 2023-10-18T04:17:22.368 INFO:tasks.workunit.client.0.smithi003.stdout: write object-1004 2023-10-18T04:17:22.368 INFO:tasks.workunit.client.0.smithi003.stdout: write object-1019 2023-10-18T04:17:22.368 INFO:tasks.workunit.client.0.smithi003.stdout: write object-1021 2023-10-18T04:17:22.369 INFO:tasks.workunit.client.0.smithi003.stdout:verify written data by reading 2023-10-18T04:17:22.369 INFO:tasks.workunit.client.0.smithi003.stdout: read object-11 2023-10-18T04:17:22.369 INFO:tasks.workunit.client.0.smithi003.stdout:made snapshot "image39@parent_snap" and protect it 2023-10-18T04:17:22.369 INFO:tasks.workunit.client.0.smithi003.stdout:made and opened clone "image40" 2023-10-18T04:17:22.369 INFO:tasks.workunit.client.0.smithi003.stdout:flattening clone: "image40" 2023-10-18T04:17:22.370 INFO:tasks.workunit.client.0.smithi003.stdout:check whether child image has the same set of objects as parent 2023-10-18T04:17:22.370 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.FlattenNoEmptyObjects (5702 ms) 2023-10-18T04:17:22.370 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.PoolMetadataConfApply 2023-10-18T04:17:23.825 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.PoolMetadataConfApply (1459 ms) 2023-10-18T04:17:23.825 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.Sparsify 2023-10-18T04:17:23.964 INFO:tasks.workunit.client.0.smithi003.stdout:sparsify_supported=1 2023-10-18T04:17:23.965 INFO:tasks.workunit.client.0.smithi003.stdout:sparse_read_supported=1 2023-10-18T04:17:24.214 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.Sparsify (389 ms) 2023-10-18T04:17:24.215 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.SparsifyClone 2023-10-18T04:17:24.330 INFO:tasks.workunit.client.0.smithi003.stdout:sparsify_supported=1 2023-10-18T04:17:28.142 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.SparsifyClone (3928 ms) 2023-10-18T04:17:28.143 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestInternal.MissingDataPool 2023-10-18T04:17:31.093 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestInternal.MissingDataPool (2951 ms) 2023-10-18T04:17:31.408 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 34 tests from TestInternal (73098 ms total) 2023-10-18T04:17:31.408 INFO:tasks.workunit.client.0.smithi003.stdout: 2023-10-18T04:17:31.409 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2023-10-18T04:17:31.409 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 34 tests from 1 test suite ran. (75363 ms total) 2023-10-18T04:17:31.409 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 29 tests. 2023-10-18T04:17:31.409 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] 5 tests, listed below: 2023-10-18T04:17:31.409 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.WriteFailsToLockImageBlocklisted 2023-10-18T04:17:31.410 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.WriteFailsToLockImageBlocklistedWatch 2023-10-18T04:17:31.410 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.SnapshotCopyup 2023-10-18T04:17:31.410 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.DiscardCopyup 2023-10-18T04:17:31.410 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestInternal.TestCoR 2023-10-18T04:17:31.468 INFO:tasks.workunit.client.0.smithi003.stdout:seed 146417 2023-10-18T04:17:31.469 INFO:tasks.workunit.client.0.smithi003.stdout:Note: Google Test filter = TestMirroring.* 2023-10-18T04:17:31.469 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 51 tests from 1 test suite. 2023-10-18T04:17:31.469 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2023-10-18T04:17:31.469 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 51 tests from TestMirroring 2023-10-18T04:17:33.418 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage 2023-10-18T04:17:50.332 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage (16912 ms) 2023-10-18T04:17:50.333 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool 2023-10-18T04:17:51.122 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool (792 ms) 2023-10-18T04:17:51.123 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled 2023-10-18T04:17:51.721 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled (599 ms) 2023-10-18T04:17:51.722 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage 2023-10-18T04:17:52.510 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage (788 ms) 2023-10-18T04:17:52.510 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap 2023-10-18T04:17:53.226 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap (716 ms) 2023-10-18T04:17:53.226 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModePool 2023-10-18T04:17:53.970 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModePool (744 ms) 2023-10-18T04:17:53.970 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled 2023-10-18T04:17:54.843 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled (873 ms) 2023-10-18T04:17:54.843 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.DisableImageMirrorWithPeer 2023-10-18T04:17:54.843 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_mirroring.cc:482: Skipped 2023-10-18T04:17:54.843 INFO:tasks.workunit.client.0.smithi003.stdout:Skipping due to unmet REQUIRE 2023-10-18T04:17:54.844 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestMirroring.DisableImageMirrorWithPeer (0 ms) 2023-10-18T04:17:54.844 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.DisableJournalingWithPeer 2023-10-18T04:17:54.844 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-6794-gfbf795c8/rpm/el9/BUILD/ceph-18.0.0-6794-gfbf795c8/src/test/librbd/test_mirroring.cc:520: Skipped 2023-10-18T04:17:54.844 INFO:tasks.workunit.client.0.smithi003.stdout:Skipping due to unmet REQUIRE 2023-10-18T04:17:54.844 INFO:tasks.workunit.client.0.smithi003.stdout:[ SKIPPED ] TestMirroring.DisableJournalingWithPeer (0 ms) 2023-10-18T04:17:54.844 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling 2023-10-18T04:17:55.347 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling (505 ms) 2023-10-18T04:17:55.348 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling 2023-10-18T04:17:55.932 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling (584 ms) 2023-10-18T04:17:55.932 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling 2023-10-18T04:18:01.272 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling (5341 ms) 2023-10-18T04:18:01.273 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock 2023-10-18T04:18:05.320 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock (4048 ms) 2023-10-18T04:18:05.321 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeDisabled 2023-10-18T04:18:05.983 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeDisabled (662 ms) 2023-10-18T04:18:05.983 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage 2023-10-18T04:18:06.661 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage (678 ms) 2023-10-18T04:18:06.662 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool 2023-10-18T04:18:08.180 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool (1519 ms) 2023-10-18T04:18:08.180 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling 2023-10-18T04:18:09.088 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling (907 ms) 2023-10-18T04:18:09.088 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling 2023-10-18T04:18:09.676 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling (589 ms) 2023-10-18T04:18:09.676 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeDisabled 2023-10-18T04:18:10.403 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeDisabled (726 ms) 2023-10-18T04:18:10.403 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage 2023-10-18T04:18:11.156 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage (753 ms) 2023-10-18T04:18:11.156 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled 2023-10-18T04:18:16.358 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled (5203 ms) 2023-10-18T04:18:16.359 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModePool 2023-10-18T04:18:17.181 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModePool (822 ms) 2023-10-18T04:18:17.181 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModePool 2023-10-18T04:18:19.045 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModePool (1865 ms) 2023-10-18T04:18:19.046 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModeImage 2023-10-18T04:18:19.802 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModeImage (757 ms) 2023-10-18T04:18:19.803 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode 2023-10-18T04:18:20.945 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode (1143 ms) 2023-10-18T04:18:20.945 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode 2023-10-18T04:18:22.097 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode (1151 ms) 2023-10-18T04:18:22.097 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode 2023-10-18T04:18:23.117 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode (1021 ms) 2023-10-18T04:18:23.117 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode 2023-10-18T04:18:24.188 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode (1070 ms) 2023-10-18T04:18:24.188 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode 2023-10-18T04:18:25.300 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode (1112 ms) 2023-10-18T04:18:25.300 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode 2023-10-18T04:18:26.605 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode (1306 ms) 2023-10-18T04:18:26.605 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageEnabled 2023-10-18T04:18:27.244 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageEnabled (638 ms) 2023-10-18T04:18:27.245 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDisabled 2023-10-18T04:18:27.844 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDisabled (601 ms) 2023-10-18T04:18:27.845 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.RemoveImage_With_ImageWithoutJournal 2023-10-18T04:18:28.310 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.RemoveImage_With_ImageWithoutJournal (465 ms) 2023-10-18T04:18:28.310 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDemoted 2023-10-18T04:18:29.285 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDemoted (975 ms) 2023-10-18T04:18:29.285 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.TrashMoveRestore_PoolMode 2023-10-18T04:18:30.216 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.TrashMoveRestore_PoolMode (932 ms) 2023-10-18T04:18:30.217 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled 2023-10-18T04:18:31.022 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled (805 ms) 2023-10-18T04:18:31.022 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled 2023-10-18T04:18:31.881 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled (859 ms) 2023-10-18T04:18:31.881 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.MirrorStatusList 2023-10-18T04:18:32.398 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.MirrorStatusList (518 ms) 2023-10-18T04:18:32.399 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.RemoveBootstrapped 2023-10-18T04:18:33.094 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.RemoveBootstrapped (695 ms) 2023-10-18T04:18:33.094 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.AioPromoteDemote 2023-10-18T04:18:37.366 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.AioPromoteDemote (4273 ms) 2023-10-18T04:18:37.367 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.AioGetInfo 2023-10-18T04:18:39.921 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.AioGetInfo (2555 ms) 2023-10-18T04:18:39.922 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.AioGetStatus 2023-10-18T04:18:41.505 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.AioGetStatus (1583 ms) 2023-10-18T04:18:41.505 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.SiteName 2023-10-18T04:18:41.514 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestMirroring.SiteName (10 ms) 2023-10-18T04:18:41.514 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestMirroring.Bootstrap 2023-10-18T04:18:50.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-18T04:18:55.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2023-10-18T04:19:00.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2023-10-18T04:19:06.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2023-10-18T04:19:11.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2023-10-18T04:19:17.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~27s 2023-10-18T04:19:22.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~32s 2023-10-18T04:19:27.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~38s 2023-10-18T04:19:33.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~43s 2023-10-18T04:19:38.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~49s 2023-10-18T04:19:44.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~54s 2023-10-18T04:19:49.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~59s 2023-10-18T04:19:54.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~65s 2023-10-18T04:20:00.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~70s 2023-10-18T04:20:05.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~76s 2023-10-18T04:20:11.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~81s 2023-10-18T04:20:16.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~86s 2023-10-18T04:20:21.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~92s 2023-10-18T04:20:27.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~97s 2023-10-18T04:20:32.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~103s 2023-10-18T04:20:38.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~108s 2023-10-18T04:20:43.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~113s 2023-10-18T04:20:49.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~119s 2023-10-18T04:20:54.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~124s 2023-10-18T04:20:59.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2023-10-18T04:21:05.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2023-10-18T04:21:10.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~140s 2023-10-18T04:21:16.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~146s 2023-10-18T04:21:21.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~151s 2023-10-18T04:21:26.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~157s 2023-10-18T04:21:32.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~162s 2023-10-18T04:21:37.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~168s 2023-10-18T04:21:43.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~173s 2023-10-18T04:21:48.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~178s 2023-10-18T04:21:53.840 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~184s 2023-10-18T04:21:59.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~189s 2023-10-18T04:22:04.647 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~195s 2023-10-18T04:22:10.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~200s 2023-10-18T04:22:15.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~205s 2023-10-18T04:22:20.857 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~211s 2023-10-18T04:22:26.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~216s 2023-10-18T04:22:31.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~222s 2023-10-18T04:22:37.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~227s 2023-10-18T04:22:42.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~232s 2023-10-18T04:22:47.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~238s 2023-10-18T04:22:53.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~243s 2023-10-18T04:22:58.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~249s 2023-10-18T04:23:04.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~254s 2023-10-18T04:23:09.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~259s 2023-10-18T04:23:14.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2023-10-18T04:23:20.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2023-10-18T04:23:25.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~276s 2023-10-18T04:23:31.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2023-10-18T04:23:36.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2023-10-18T04:23:41.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2023-10-18T04:23:47.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2023-10-18T04:23:52.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~303s 2023-10-18T04:23:52.712 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-10-18T04:23:53.114 INFO:tasks.ceph.osd.0:Sent signal 15 2023-10-18T04:23:53.115 INFO:tasks.ceph.osd.1:Sent signal 15 2023-10-18T04:23:53.115 INFO:tasks.ceph.mon.a:Sent signal 15 2023-10-18T04:23:53.116 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-10-18T04:23:53.117 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-18T04:23:53.114+0000 7f6a7e7fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 83294) UID: 0 2023-10-18T04:23:53.118 INFO:tasks.ceph.mon.a.smithi003.stderr:2023-10-18T04:23:53.114+0000 7f6a7e7fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-18T04:23:53.118 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T04:23:53.114+0000 7f99fcff7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 83296) UID: 0 2023-10-18T04:23:53.119 INFO:tasks.ceph.mgr.x.smithi003.stderr:2023-10-18T04:23:53.114+0000 7f99fcff7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-10-18T06:44:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-18T06:44:46.232 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2023-10-18T06:44:46.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-18T06:44:46.536 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi003 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf795c854382dc4ab29c95372096a3ba925ed95 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2023-10-18T06:44:46.608 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=28a284e865f84f4f88a2427cac8cd2df Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi003 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf795c854382dc4ab29c95372096a3ba925ed95 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2023-10-18T06:44:46.611 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-18T06:44:46.621 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-18T06:44:46.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-18T06:44:46.746 INFO:teuthology.orchestra.run.smithi003.stderr:2023-10-18T06:44:46.740+0000 7fc969111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-18T06:46:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-18T06:46:46.673 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-18T06:46:46.682 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-18T06:46:46.682 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-18T06:46:46.683 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-18T06:46:46.683 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T06:46:52.786 INFO:tasks.ceph.osd.0:Stopped 2023-10-18T06:46:52.787 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-18T06:46:52.787 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T06:46:52.881 INFO:tasks.ceph.osd.1:Stopped 2023-10-18T06:46:52.882 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-18T06:46:52.883 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T06:46:52.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T06:46:52.884 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2023-10-18T06:46:52.895 INFO:tasks.ceph.osd.2:Stopped 2023-10-18T06:46:52.895 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-18T06:46:52.896 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-18T06:46:52.896 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T06:46:52.899 INFO:tasks.ceph.mgr.x:Stopped 2023-10-18T06:46:52.900 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-18T06:46:52.900 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-18T06:46:52.901 INFO:teuthology.orchestra.run:waiting for 300 2023-10-18T06:46:52.902 INFO:tasks.ceph.mon.a:Stopped 2023-10-18T06:46:52.902 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-18T06:46:52.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(POOL_FULL\)' | head -n 1 2023-10-18T06:46:52.940 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T06:46:52.941 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-18T06:46:53.099 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T06:46:53.100 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-18T06:46:53.165 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T06:46:53.166 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-18T06:46:53.240 INFO:tasks.ceph:Archiving mon data... 2023-10-18T06:46:53.242 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264/data/mon.a.tgz 2023-10-18T06:46:53.243 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-10-18T06:46:53.258 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.TmgdpH37Kd 2023-10-18T06:46:53.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TmgdpH37Kd 2023-10-18T06:46:53.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.TmgdpH37Kd 2023-10-18T06:46:53.492 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.TmgdpH37Kd is 309KB 2023-10-18T06:46:53.588 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.TmgdpH37Kd 2023-10-18T06:46:53.604 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-18T06:46:53.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-18T06:46:53.673 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-18T06:46:53.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-18T06:46:53.746 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-18T06:46:53.750 INFO:tasks.ceph:Archiving crash dumps... 2023-10-18T06:46:53.753 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264/remote/smithi003/crash 2023-10-18T06:46:53.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-18T06:46:53.820 INFO:tasks.ceph:Compressing logs... 2023-10-18T06:46:53.821 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-18T06:46:53.893 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83764.log 2023-10-18T06:46:53.893 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-18T06:46:53.893 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85218.log 2023-10-18T06:46:53.894 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.83764.log: /var/log/ceph/ceph-mon.a.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83764.log.gz 2023-10-18T06:46:53.894 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83386.log 2023-10-18T06:46:53.894 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124471.log 2023-10-18T06:46:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85218.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85218.log.gz 2023-10-18T06:46:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.83386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85176.log 2023-10-18T06:46:53.895 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83386.log.gz 2023-10-18T06:46:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.124471.log: -- /var/log/ceph/ceph-client.admin.83496.log 2023-10-18T06:46:53.909 INFO:teuthology.orchestra.run.smithi003.stderr: 80.0% -- replaced with /var/log/ceph/ceph-client.admin.124471.log.gz 2023-10-18T06:46:53.910 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115609.log 2023-10-18T06:46:53.910 INFO:teuthology.orchestra.run.smithi003.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85176.log.gz 2023-10-18T06:46:53.910 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.83496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83536.log 2023-10-18T06:46:53.910 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83496.log.gz 2023-10-18T06:46:53.911 INFO:teuthology.orchestra.run.smithi003.stderr:gzip/var/log/ceph/ceph-client.admin.115609.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.98984.log 2023-10-18T06:46:53.911 INFO:teuthology.orchestra.run.smithi003.stderr: 80.5% -- replaced with /var/log/ceph/ceph-client.admin.115609.log.gz 2023-10-18T06:46:53.911 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.83536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85384.log 2023-10-18T06:46:53.912 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83536.log.gz 2023-10-18T06:46:53.912 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.98984.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-18T06:46:53.912 INFO:teuthology.orchestra.run.smithi003.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.98984.log.gz 2023-10-18T06:46:53.912 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.85384.log: /var/log/ceph/ceph-client.admin.106858.log 2023-10-18T06:46:53.913 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85384.log.gz 2023-10-18T06:46:53.913 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85059.log 2023-10-18T06:46:53.913 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.106858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99161.log 2023-10-18T06:46:53.914 INFO:teuthology.orchestra.run.smithi003.stderr: 80.2% -- replaced with /var/log/ceph/ceph-client.admin.106858.log.gz 2023-10-18T06:46:53.914 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85510.log 2023-10-18T06:46:53.914 INFO:teuthology.orchestra.run.smithi003.stderr: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-18T06:46:53.914 INFO:teuthology.orchestra.run.smithi003.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85059.log.gz 2023-10-18T06:46:53.915 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.99161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83576.log 2023-10-18T06:46:53.915 INFO:teuthology.orchestra.run.smithi003.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.99161.log.gz 2023-10-18T06:46:53.915 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85510.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80049.log 2023-10-18T06:46:53.915 INFO:teuthology.orchestra.run.smithi003.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85510.log.gz 2023-10-18T06:46:53.916 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.83576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85382.log 2023-10-18T06:46:53.916 INFO:teuthology.orchestra.run.smithi003.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83576.log.gz 2023-10-18T06:46:53.916 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.tmp-client.admin.80049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85116.log 2023-10-18T06:46:53.917 INFO:teuthology.orchestra.run.smithi003.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80049.log.gz 2023-10-18T06:46:53.917 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146417.log 2023-10-18T06:46:53.917 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85382.log.gz 2023-10-18T06:46:53.917 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85302.log 2023-10-18T06:46:53.918 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85116.log.gz 2023-10-18T06:46:53.918 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.146417.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-18T06:46:53.918 INFO:teuthology.orchestra.run.smithi003.stderr: 80.1% -- replaced with /var/log/ceph/ceph-client.admin.146417.log.gz 2023-10-18T06:46:53.919 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85898.log 2023-10-18T06:46:53.919 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85302.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85302.log.gz 2023-10-18T06:46:53.919 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106789.log 2023-10-18T06:46:53.919 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.85898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146363.log 2023-10-18T06:46:53.920 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.106789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85383.log 2023-10-18T06:46:53.920 INFO:teuthology.orchestra.run.smithi003.stderr: 80.3% -- replaced with /var/log/ceph/ceph-client.admin.106789.log.gz 2023-10-18T06:46:53.920 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.146363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85260.log 2023-10-18T06:46:53.920 INFO:teuthology.orchestra.run.smithi003.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.85898.log.gz 2023-10-18T06:46:53.921 INFO:teuthology.orchestra.run.smithi003.stderr: 81.1% -- replaced with /var/log/ceph/ceph-client.admin.146363.log.gz 2023-10-18T06:46:53.921 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147044.log 2023-10-18T06:46:53.921 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85383.log.gz 2023-10-18T06:46:53.922 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85596.log 2023-10-18T06:46:53.922 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85260.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85260.log.gz 2023-10-18T06:46:53.922 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.147044.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-18T06:46:53.922 INFO:teuthology.orchestra.run.smithi003.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.147044.log.gz 2023-10-18T06:46:53.923 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85508.log 2023-10-18T06:46:53.923 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85596.log.gz 2023-10-18T06:46:53.923 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85680.log 2023-10-18T06:46:53.924 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85508.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-18T06:46:53.924 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85508.log.gz 2023-10-18T06:46:53.924 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83298.log 2023-10-18T06:46:53.924 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85680.log.gz 2023-10-18T06:46:53.925 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83734.log 2023-10-18T06:46:53.925 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.83298.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-18T06:46:53.925 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83298.log.gz 2023-10-18T06:46:53.925 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.83734.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-18T06:46:53.926 INFO:teuthology.orchestra.run.smithi003.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83734.log.gz 2023-10-18T06:46:53.926 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85638.log 2023-10-18T06:46:53.927 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146297.log 2023-10-18T06:46:53.927 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.85638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106925.log 2023-10-18T06:46:53.927 INFO:teuthology.orchestra.run.smithi003.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85638.log.gz 2023-10-18T06:46:53.927 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.146297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138621.log 2023-10-18T06:46:53.928 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.106925.log: 76.7% -- replaced with /var/log/ceph/ceph-client.admin.146297.log.gz 2023-10-18T06:46:53.928 INFO:teuthology.orchestra.run.smithi003.stderr: 80.3% -- replaced with /var/log/ceph/ceph-client.admin.106925.log.gz 2023-10-18T06:46:53.928 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85509.log 2023-10-18T06:46:53.928 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.138621.log: /var/log/ceph/ceph-client.admin.85509.log: 80.2% -- replaced with /var/log/ceph/ceph-client.admin.138621.log.gz 2023-10-18T06:46:53.929 INFO:teuthology.orchestra.run.smithi003.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85509.log.gz 2023-10-18T06:46:53.929 INFO:teuthology.orchestra.run.smithi003.stderr: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-18T06:46:54.074 INFO:teuthology.orchestra.run.smithi003.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-18T06:46:54.705 INFO:teuthology.orchestra.run.smithi003.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-18T06:47:03.141 INFO:teuthology.orchestra.run.smithi003.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-18T06:47:03.167 INFO:teuthology.orchestra.run.smithi003.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-18T06:47:04.930 INFO:teuthology.orchestra.run.smithi003.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-18T06:47:04.931 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-10-18T06:47:04.931 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m11.053s 2023-10-18T06:47:04.931 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m29.320s 2023-10-18T06:47:04.932 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m1.213s 2023-10-18T06:47:04.932 INFO:tasks.ceph:Archiving logs... 2023-10-18T06:47:04.933 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264/remote/smithi003/log 2023-10-18T06:47:04.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-18T06:47:08.877 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-18T06:47:08.887 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-18T06:47:08.897 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-18T06:47:08.902 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-18T06:47:08.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-18T06:47:08.969 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-18T06:47:08.978 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-18T06:47:08.979 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-18T06:47:08.999 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2023-10-18T06:47:09.003 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-18T06:47:09.004 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-10-18T06:47:09.004 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv01.front.sepia.ceph.com 3 7 377 90 +2590us[+2590us] +/- 67ms 2023-10-18T06:47:09.004 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv02.front.sepia.ceph.com 3 9 377 273 -2228us[-2269us] +/- 65ms 2023-10-18T06:47:09.005 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 98 -703us[ -703us] +/- 63ms 2023-10-18T06:47:09.005 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-18T06:47:09.006 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T06:47:09.015 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T06:47:09.016 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T06:47:09.025 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T06:47:09.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T06:47:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T06:47:09.189 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T06:47:09.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T06:47:09.207 INFO:teuthology.task.internal:Duration was 11544.516399 seconds 2023-10-18T06:47:09.207 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T06:47:09.216 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T06:47:09.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T06:47:09.279 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T06:47:09.432 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T06:47:09.432 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2023-10-18T06:47:09.433 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T06:47:09.453 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T06:47:09.454 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T06:47:09.524 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T06:47:09.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T06:47:09.668 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T06:47:09.679 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T06:47:09.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T06:47:09.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T06:47:09.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T06:47:09.779 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2023-10-18T06:47:09.803 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T06:47:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T06:47:09.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T06:47:09.868 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T06:47:09.869 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264/remote/smithi003 2023-10-18T06:47:09.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T06:47:09.937 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T06:47:09.938 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T06:47:09.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T06:47:10.001 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T06:47:10.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T06:47:10.014 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T06:47:10.015 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T06:47:10.041 INFO:teuthology.orchestra.run.smithi003.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 18 06:47 /home/ubuntu/cephtest 2023-10-18T06:47:10.042 INFO:teuthology.orchestra.run.smithi003.stdout: 402801 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 18 03:44 /home/ubuntu/cephtest/mnt.0 2023-10-18T06:47:10.042 INFO:teuthology.orchestra.run.smithi003.stdout: 402802 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 18 03:44 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-18T06:47:10.043 INFO:teuthology.orchestra.run.smithi003.stdout: 538124 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 03:44 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-18T06:47:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T06:47:10.044 INFO:teuthology.orchestra.run.smithi003.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-18T06:47:10.045 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_fbf795c854382dc4ab29c95372096a3ba925ed95/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-18T06:47:10.058 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T06:47:10.070 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T06:47:10.080 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi003 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-18T06:47:10.081 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T06:47:10.109 DEBUG:teuthology.nuke:shortname: smithi003 2023-10-18T06:47:10.110 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T06:47:10.132 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2023-10-18_03:24:35-crimson-rados-wip-crimson-scrub-testing-2023-10-17-distro-default-smithi/7431264', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 03:26:29.627168', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5XHdjHt2RFShvGSuGm1VMFOoHPjtSMZgUHDtXvvx4lESLwrSniIqeicsTbOvTtgUbPojTArnahW1WOutSZ4tw='} 2023-10-18T06:47:10.186 INFO:teuthology.orchestra.console.smithi003:Power off 2023-10-18T06:47:10.187 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T06:47:10.211 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2023-10-18T06:47:10.211 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T06:47:10.234 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-10-18T06:47:14.236 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T06:47:14.350 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-10-18T06:47:18.354 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T06:47:18.465 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2023-10-18T06:47:18.566 INFO:teuthology.orchestra.console.smithi003:Power off completed 2023-10-18T06:47:18.672 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} duration: 11544.516399383545 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi003 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf795c854382dc4ab29c95372096a3ba925ed95 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_sjust@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=28a284e865f84f4f88a2427cac8cd2df status: fail success: false 2023-10-18T06:47:18.673 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T06:47:18.753 INFO:teuthology.run:FAIL